The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good

Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn

Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US

July 2016

The Blueprint for Radically Improved Training Evaluations

“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to

improve the value of training.” (p. 94‐95)

Tim Mooney and Rob BrinkerhoffCourageous Training:

Bold Actions for Business Results.

Demonstrate the Value

Improve theLearning

Three Reasons to Use Measurement in Learning

# 1

# 2

# 3

The Kirkpatrick 4-Level ModelLevel 1

LearnerReaction

Level 2

Tests of Learning

Level 3

On‐the‐Job Behavior

Level 4

Business Results

“Training Evaluations”

reaction sheetshappy sheets

level 1’s

Bridging Gap between Research and Practice

Research

Simplicity is the Ultimate Sophistication

Leonardo da Vinci

The Decisive Dozenfor Learning Design and Learning Measurement

http://is.gd/ddResearch

1. Content

2. ExposureBaseline

3. Guiding Attention

4. Creating Correct Conceptions

5. Repetition

6. Feedback

7. Variation

Engagement & Understanding

8. Retrieval Practice

9. Context Alignment

10. SpacingRemembering

11. Persuasion

12. PerseveranceApplication

http://Work-Learning.com/catalog.html

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan

University & Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

Background

This year, improving your smile sheetswill likely be the MOST IMPORTANTthing you can do to improve learning!

Professional Development

Pro

1. Red‐flagging training programs that are not sufficiently effective.

2. Gathering ideas for ongoing updates and revision of a learning program.

3. Judging strengths and weaknesses of a pilot program to enable revision.

4. Providing instructors with feedback to aid their development.

5. Helping learners reflect on and reinforce what they learned.

6. Helping learners determine what (if anything) they plan to do with their learning.

7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.

8. Upholding the spirit of common courtesy by giving learners a chance for feedback.

9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.

Modified based on work by Robert Brinkerhoff

Reasons for Smile Sheets

# 1

We Work in aLearning-to-Performance

Ecosystem

LearningIntervention

PerformanceSituation

LearningOutcomes

IndividualResults

OrganizationalResults

On-the-Job Learning

Prompting

The Learning Landscape

Learning

Remembering

On-the-Job Performance

Later on the job, learners remember

what they’ve learned.

The learners get a return on their efforts.

Performance can be prompted

through job aids, signage,

intuitive cues, performance

support, management, etc.

Learners can learn on-the-job through

trial & practice, insight learning, help from others,

social media, studying on their

own, etc.

© Copyright 2009-2016 Work-Learning Research, Inc.

YouTube: http://is.gd/LearningLandscape

# 2

We Should NOTExpect Perfection

20

Measurement doesn’t

illuminate everything.

Measurement

Measurement illuminates

some things.

There is no perfect measurement tool!

How would you rate your organization’s smile sheets?

Terrible

1

Bad

2

Okay

3

Good

4

Given, that there is NO perfect

measurement…

# 3

The Kirkpatrick ModelControls our Thinking!

The Kirkpatrick 4-Level ModelLevel 1

LearnerReaction

Level 2

Tests of Learning

Level 3

On‐the‐Job Behavior

Level 4

Business Results



LearningOutcomes

IndividualResults


On-the-Job Learning

Prompting


Learning

Remembering


© Copyright 2009-2016 Work-Learning Research, Inc.Regular

smile sheets

Immediate tests of

remembering & decision-

making

Does Job Performance

Improve?

Does Business

Performance Improve?

IR

L1-Delayed

L2-Delayed

L3-Supports

PD

RS JS PS

L1

L2Are Prompt Supports

Used? Effective?

Are Just-in-Time Learning

Supports Used?

Effective?

Are Retrieval Supports

Used? Effective?

Smile sheets

Retrieval &

Decision Making

Evaluation of Coaching & Learner Learning

Delayed smile

sheets

Delayed tests of remembering & decision-making

Do the learners actually benefit

from the learning in their work or

career?

Does the work environment

support performance?

Are Prompting Devices Used?

Effective?

On smile sheets, ask about

motivation to apply

L1

L2

L3

L4

Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation

• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.

• Training centric. Ignores prompting mechanisms & on-the-job learning.

• Ignores the role that management and the business side must play.

• Implies that higher levels are more important than lower levels.

• Ignores the causal chain from learning to remembering to performance to results.

• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.

• Pushes us to value learner ratings as predictive of learning and on-the-job performance.

“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…

[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.

[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)

http://is.gd/TrainingResearch2012

# 4

Most Smile Sheets Stink!

Level 1to

Level 3

r=.16

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Very Weak Relationship between Levels

Correlation between levels?

Level 1to

Level 2

r=.09

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied

Psychology, 93, 280-295.

Correlation between levels?

No

Practical Significance

Weak Relationship is below .30 and .09 is VERY WEAK

So…SMILE SHEETS tell us VERY LITTLE about Learning

Level 1to

Level 2

r=.09

# 5

Likert-Like Scales &Numeric Responses

create Poor Data

Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree

54321

4.1

Compared to:

• Previous• Standard• Others

Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:

On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):

“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)

We’d like to trust our learners…

Research shows that learnersdon’t always know their own learning…

Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).

Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).

Learners Fail to Properly Use ExamplesRenkl (1997).

Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).

Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).

Transmogrify

Strongly AgreeAgree

Neither Agree Nor DisagreeDisagree

Strongly Disagree

We Start with Fuzzy

Adjectives

54321

Magically We Turn Adjectives

Into Numbers

We Average

Responses, Losing

More Info

3.9

4.2

3.7

4.0

4.1

4.0

4.2

3.4

We Choose One

Question and Report Results

4.1

Garbage In -- Garbage Out!

When Our Smile Sheet Questions areBiased, Irrelevant, Unanswerable

Garbage In -- Garbage Out!

# 6

The Critical Importanceof Remembering!



LearningOutcomes

IndividualResults




Remembering

Learning

© Copyright 2009-2016 Work-Learning Research, Inc., www.work-learning.com

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

After LearningDuring Learning

Learning and Forgetting Curves

Learners

Learners

If our learners start here.

But end up here.

Have we maximized the

learning benefits?

Less Remembering

More Remembering

© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15


On-the-JobLearning Curves

On-the-JobForgetting

Curves

LearningCurve


There are many possible after-training results,

depending on:

(1) Design of the Learning(2) After-Learning Follow-up


YouTube: http://is.gd/LearningForgettingCurves

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15


On-the-JobLearning Curves

On-the-JobForgetting

Curves

LearningCurve



What does an end-of-course

assessment tell us?

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




If we measure

here?BiasedMetric!!

Understanding

Remembering

Application

DelayedTests

# 7

We must go beyondSmile Sheets!

Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts

Smile Sheets

Understanding

Remembering

Motivation to Apply

After Supports

Meeting Target Goals?

Job Performance

Organizational Results

Learner Expectations

Other Expectations

Supports:

Management Support?

Workplace Obstacles?

Reinforcement?

Reminders?

Learners able to:

Understand?

Remember?

Make Decisions?

Apply the Learning?

# 8

Smile Sheetscan be Improved!

My Journey in tryingTo create a better

Smile Sheet

My “New” Smile Sheet

Specific Concepts

How much Value?

ConceptNew?

HowWell

Taught?

Simple Overall Ratings

Helping Learners Calibrate

Best Feedback Comes from Comments

# 9

Performance-FocusedSmile Sheets

http://is.gd/TrainingResearch2012

http://is.gd/ddResearch

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNINGEFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding

numeric averages that discourage a standards-based decision on success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at

least try to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

Three Key Goals:

1. Are the Questions Well-Designed?

2. Are Learners Making Good Smile Sheet Decisions?

3. Are The Data we’re Getting Clear and Actionable?

Ensuring that we are focused on the science-

of-learning factors that matter!

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNING EFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding

numeric averages that discourage a standards-based decision on success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at

least try to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

What Most Smile Sheets

Provide

QUATERNARYGOALS

Traditional Smile Sheets

Learners rate instructors as credible and engaging?

Learners say that classroom environment was conducive to

learning?

Learners satisfied with experience?

Learners think course was well-organized?

Ultimate Goal Primary Goals Secondary Goals Tertiary Goals

In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?

A. I’m NOT AT ALL ABLE to put the concepts into practice.

B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.

C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.

D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.

E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.

A Better Smile Sheet Question


HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.

A. I am NOT AT ALL ready to use the skills taught.

B. I need MORE GUIDANCE to be GOOD at using these skills.

C. I need MORE EXPERIENCE to be GOOD at using these skills.

D. I am FULLY COMPETENT in using these skills.

E. I am CAPABLE at an EXPERT LEVEL in using these skills.

A Better Smile Sheet Question -- Shorter

© Copyright 2009 Work-Learning Research, Inc.

Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?

A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.

B. I have a BASIC FAMILIARITY with the concepts.

C. I have a SOLID UNDERSTANDING of the concepts.

D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.

E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.

Standards

Unacceptable

Unlikely

?

Acceptable

?

AcceptableUnacceptable

AcceptableSuperior

Traditional Smile Sheet

Design Learning

Design Smile Sheet

Deploy‐‐‐‐‐

Collect Data

Examine Results

Determine Standards

Performance‐Focused Smile

Sheet

Design Learning

Design Smile Sheet

Deploy‐‐‐‐‐

Collect Data

Examine Results

Determine Standards

OPEN TO

BIAS


In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?

A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.

B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.

C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job.

D. I will make this a HIGH PRIORITY when I get back to my day-to-day job.

E. I will make this one of my HIGHEST PRIORITIES when I get back to my day-to-day job.

A Question About Motivation…


After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.

A. I will have ENOUGH TIME to work on applying the learning.

B. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.

C. I will have someone available TO COACH OR MENTOR ME in applying the learning.

D. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.

E. I will have JOB AIDS to guide me in applying the learning to real job tasks.

F. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.

G. I will NOT get much direct support, but will rely on my own initiative.

A Question About Follow-Through…

# 10

Delayed Smile Sheets

Delayed Smile Sheet

Have you used what you learned in the workshop to make a significant improvement in your work?

• No, and I doubt that I will use what I learned.

• No, but I probably will use what I learned.

• No, but I have a plan to use what I learned.

• Yes, I have already used what I learned.

Are You Using What You Learned?

Yes!!

What Enabled You to Apply

What You Learned?

No

What is Holding You Back?

Feedback to:• Management• Learners• Us

Delayed Smile Sheet – Feedback for Improvement

What is holding you back from teaching others what you have learned?

• The learning content is not relevant to my colleagues’ work.

• I don’t think the learning content is valid.

• I don’t see how teaching others will benefit me.

• I can’t remember the learning content well enough.

• I’m not very good at teaching others.

• I haven’t had the time.

• I have had higher priorities.

• I have not had the resources.

• The risk of teaching this to others is too high.

• I’m still working to persuade management of the value.

• Other, please specify ____________________________

Delayed Smile Sheet

―――Feedback

toManagement,

Learners,and/or Us.

What are the top 3 to 4 reasons that have enabled you to put the workshop information into practice?

• I remember (or can look up) the workshop’s learning points.

• I believe strongly in the workshop’s learning points.

• I have the authority to make the necessary changes.

• I have decided to lead a change effort to make this happen.

• I have the time to put this into practice.

• I have the resources to put this into practice.

• I will be acknowledged or rewarded if I do this.

• I will be sanctioned or punished if I do NOT do this.

• My management has made this a high priority.

• My management is fully supportive of my efforts.

• My coworkers are fully supportive of my efforts.

• My Team (or our whole unit) is working together on this.

50%

50%75%

50%

Message toManagement

“More management-

driven than self-driven.”

This learning event will make a significant contribution to my work, in terms of the following: SELECT AS MANY AS YOU LIKE!

A. Improving my personal productivity.

B. Increasing my ability to innovate.

C. Enabling me to collaborate more effectively.

D. Improving my management performance.

E. Enabling me to generate more revenue.

F. Enabling me to lower costs.

# 11

Stealth Messages

Workplace Learning‐and‐Performance

Professionals

1. Training is not enough.

2. Training for awareness is a weak and insufficient approach.

3. Training should specifically minimize forgetting and support remembering.

4. It is better to train a few things really well, than many things perfunctorily.

5. Training should motivate subsequent on‐the‐job application.

6. Training content must be correct, validated, and relevant.

7. Traditional smile sheets are inadequate.

8. We must measure to provide ourselves with feedback to spur continuous improvement.

9. Prompting mechanisms (like job aids and performance support) should be used—as appropriate—as an integral part of training and as a replacement for training.

10. On‐the‐job learning should be leveraged in addition to formal training.

11. Learners’ managers and others within the learners’ chain of command should support after‐training application.

12. Supervisors play a critical role in enabling the success of training and in helping people be creative on the job.

Some Stealth MessagesWe Might Want to Send

conscious conscious

filters

ProactiveReactive

Etcetera

There are

Leverage Pointswithin our SOP’s

from where we can send

Stealth Messages

Recruiting of Work‐Learning Professionals

Formal Course Evaluations

TrainingRequests

(our response)

Instructor Evaluations

On‐the‐Job Performance

Data

Workplace Performance Assistance

Learning Needs Analysis

Annual Reports to Management

Course Reviews

Does the course use

methods that support

remembering?

Does the instructor use methods that support after‐

training follow‐through

?

Do we determine whether this is a training issue or a workplace

issue?

Do we ask trainers to show their consulting skills, not just their

platform skills?

Do we utilize Performance‐Focused smile

sheets & delayed smile

sheets?

What messages does a

traditional smile sheet

send?

What messages does a delayed performance‐focused smile sheet send?

What messages does a

performance‐focused smile sheet send?

Summary

1. Most smile sheets provide misleading feedback.

2. Smile sheets can be significantly improved.

3. Smile Sheets should NOT be our sole source of feedback.

4. Learners don’t always know their own learning.

5. Likert‐like scales and numeric scales create a garbage‐in garbage‐out problem.

6. We should set smile‐sheet standards before we get smile‐sheet results.

7. Delayed smile sheets offer potent information.

8. We are getting very poor feedback on the success of our learning interventions.

9. We should utilize the science of learning in our learning metrics.

10. To educate our stakeholders, we should consider using stealth messaging.

11. Performance‐Focused Smile Sheets align with science‐of‐learning findings and enable better decision‐making and reporting that enables urgency and clarity.

12. The Kirkpatrick Model is incomplete and sends some wrong messages about learning measurement.

Today’s Main Points:

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan

University & Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

www.SmileSheets.comuse Code: NGO16

• New Smile‐Sheet Questions – https://is.gd/NewSmiles1

• Avoid the Net Promoter Score — http://is.gd/NetPromoter

• Avoid Benchmarking — http://is.gd/DoNotBenchmark

• eLearning Manifesto 22 Principles — http://is.gd/manifesto22

• ATD Article on Smile Sheets — http://is.gd/atdsmile

• Subscription Learning — http://SubscriptionLearning.com

• Neuroscience & Learning: Not Yet! — http://is.gd/brainlearning

Subscribe to my Newsletter: — http://Work‐Learning.com/sign‐up.html

Slides available at: www.is.gd/willstuff

Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn

Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US

July 2016

The Blueprint for Radically Improved Training Evaluations

www.SmileSheets.comuse Code: NGO16

Slides available at: www.is.gd/willstuff

Thank you!!

Documents

The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good