Phillips Associates Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Learn 11 Surprising Techniques for Obtaining Powerful

Phillips Associates

Learn 11 Surprising Learn 11 Surprising

Techniques for Obtaining Techniques for Obtaining

Powerful Data From Powerful Data From

Level 1 Evaluations Level 1 Evaluations

Learn 11 Surprising Learn 11 Surprising

Techniques for Obtaining Techniques for Obtaining

Powerful Data From Powerful Data From

Level 1 Evaluations Level 1 Evaluations Presented by:Ken PhillipsPhillips AssociatesApril 10, 2012

Phillips Associates 2

AGENDAAGENDAAGENDAAGENDA

➤ Review Kirkpatrick/Phillips five level evaluation model

➤ Examine Level 1 evaluation facts

➤ Analyze strengths and shortcomings of a sample Level 1 evaluation form

➤ Discover 11 tips to create Level 1 evaluations that produce valued data


Levels of Evaluation Measurement Focus Time Frame

Level 1: Reaction Participant reaction to a learning program

Conclusion of program

Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes

Conclusion of program or within 6 to 8 weeks after

Level 3: Behavior Degree to which participants applied back-on-the-job what was learned

2 to 12 months

Level 4: Results Degree to which targeted business outcomes were achieved

9 to 18 months

Level 5: ROI Degree to which monetary program benefits exceed program costs

9 to 18 months

KIRKPATRICK/PHILLIPS EVALUATION KIRKPATRICK/PHILLIPS EVALUATION MODELMODEL

KIRKPATRICK/PHILLIPS EVALUATION KIRKPATRICK/PHILLIPS EVALUATION MODELMODEL


LEVEL 1 FACTS*LEVEL 1 FACTS*

➤ 92% of organizations evaluate at least some learning programs at Level 1

➤ Organizations that use Level 1s on average evaluate 78% of all learning programs

➤ 36% of organizations view data collected as having high or very high value

*2009 ASTD Research Study: The Value of Evaluation: Making Training Evaluations More Effective


WHY THE DISCONNECT?WHY THE DISCONNECT?

➤ Level 1 Evaluations focus on learning department not business operation data

➤ Examples• Facility

• Course design

• Facilitator

• Food

➤ The data collected is not used


WHAT’S THE SOLUTION?WHAT’S THE SOLUTION?

➤ Create Level 1 Evaluations that produce data with high perceived value for both you & your business stakeholders

➤ Aggregate data and conduct trend lines or comparisons at group, project, department, regional level


SAMPLESAMPLE

EVALUATIONEVALUATION

FORMFORM

SAMPLESAMPLE

EVALUATIONEVALUATION

FORMFORM


11 TIPS FOR CREATING LEVEL 1 11 TIPS FOR CREATING LEVEL 1

EVALUATIONS THAT PRODUCE EVALUATIONS THAT PRODUCE

VALUED DATAVALUED DATA


TIP 1 (CONTENT)TIP 1 (CONTENT)TIP 1 (CONTENT)TIP 1 (CONTENT)

Only ask questions that lead to

actionable data and focus on

important issues


WHAT’S WRONG WITH THESE?WHAT’S WRONG WITH THESE?WHAT’S WRONG WITH THESE?WHAT’S WRONG WITH THESE?

3c. The workshop materials were well

designed

5c. The room set-up and size were

comfortable



Write learner-centered not trainer-

centered evaluation items*

*Jim Kirkpatrick, “The New World Level 1 Reaction Sheets, “unpublished article


EXAMPLESEXAMPLESEXAMPLESEXAMPLES

3c. The workshop materials aided my

understanding of the program

content

5c. My learning was enhanced by the

room set-up and size



Where appropriate, match up qualitative

questions with quantitative measures


EXAMPLEEXAMPLEEXAMPLEEXAMPLE

In a word, how would you describe thissession? ____________________________

Using a number, how would you describe this session?

No GreatValue Value

1 2 3 4 5 6 7



Include at least one item asking

participants how relevant the learning

event/material was to them and their job



This:This: How would you rate the overall relevance of this session to you and your job?

Not at all VeryRelevant Relevant

1 2 3 4 5 6 7

Not this: Not this: 3a. Overall I was satisfied with the program


TIP 5 (CONTENT)TIP 5 (CONTENT)

Create predictive questions that forecast

participant learning, intent to apply

what was learned back-on-the-job and

likely impact on business results


EXAMPLE – LEVEL 2 LEARNING EXAMPLE – LEVEL 2 LEARNING

How much did you know about developing individual performance objectives before attending this session?No ThoroughKnowledge Knowledge

1 2 3 4 5 6 7

How much do you know about developing individual performance objectives after attending the session?No ThoroughKnowledge Knowledge

1 2 3 4 5 6 7


EXAMPLE – LEVEL 3 BEHAVIOREXAMPLE – LEVEL 3 BEHAVIOR

How likely are you to apply the skills and behaviors learned in this seminar back-on-the-job?

What obstacles, if any, are likely to prevent you from applying these skills and behaviors back on the job?____________________________

Not at all Likely

Extremely Likely

1 2 3 4 5 6 7


EXAMPLE – LEVEL 4 RESULTSEXAMPLE – LEVEL 4 RESULTS

How likely are any of the key business metrics tracked by your department to improve as a result of you applying the knowledge and skills you learned in this program?Not at all

LikelyExtremely

Likely

1 2 3 4 5 6 7

How confident are you in your response?Not at all Confident

Extremely Confident

1 2 3 4 5 6 7


WHAT’S WRONG WITH THIS?WHAT’S WRONG WITH THIS?

SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree

SA A D SD


TIP 6 (MEASUREMENT)TIP 6 (MEASUREMENT)TIP 6 (MEASUREMENT)TIP 6 (MEASUREMENT)

*Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”, Harvard Business Review, 2002.

When collecting quantitative data using a

Likert scale, create a response scale

with numbers at regularly spaced

intervals and words only at each end*


Not at all True

Completely True

1 2 3 4 5 6 7


This:This:

Not This:Not This:Not at all

TrueRarelyTrue

Occasionally True

SomewhatTrue

MostlyTrue

FrequentlyTrue

Completely True

1 2 3 4 5 6 7

Never This:Never This:Not at all

TrueRarelyTrue

Occasionally True

SomewhatTrue

MostlyTrue

FrequentlyTrue

Completely True



* Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”

Use a response scale with an odd number

of points (7, 9, & 11 point scales are

best)*


ODD vs EVEN SCALEODD vs EVEN SCALEODD vs EVEN SCALEODD vs EVEN SCALE



Place small numbers at the left or low end

of the scale and large numbers at the

right or high end of the scale


EXAMPLEEXAMPLE

Not at all True

Completely True

1 2 3 4 5 6 7

Completely True

Not at all True

7 6 5 4 3 2 1

This:This:

Not This:Not This:


WHAT’S WRONG WITH THESE?WHAT’S WRONG WITH THESE?

4b. Did the facilitator clearly describe the

learning objectives?

4c. Did the facilitator keep the session lively

and interesting?



Write items either as a continuum or as a

statement


EXAMPLESEXAMPLESEXAMPLESEXAMPLES

ThisThis:: How effectively did the AV materials used during the session help reinforce your understanding of the program material?

Or ThisOr This:: The AV materials used during the session helped reinforce my understanding of the program material.Not at all

TrueCompletely

True

1 2 3 4 5 6 7

Not Effectively

Very Effectively

1 2 3 4 5 6 7


TIP 10 (ADMINISTRATION)TIP 10 (ADMINISTRATION)

Only use Level 1 evaluations to improve a

learning program not to prove

something


TIP 11 (FORMAT)TIP 11 (FORMAT)

Place questions regarding respondent

demographics (e.g. name, title,

department, etc.) at end of evaluation

form, make completion optional and

keep questions to a minimum


REFERENCESREFERENCESREFERENCESREFERENCES

Boehle, Sarah, “Remember that mean, crabby teacher inhigh school?”, Training, August 2006, pps. 17-22.

Morrel-Samuels, Palmer, “Getting the Truth intoWorkplace Surveys”, Harvard Business Review,February 2002, pps. 111-118.

Phillips, Ken, “Eight Tips on Developing Valid Level 1Evaluation Forms”, Training Today, Fall 2007,

pps. 8 & 14.

Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.


Ken Phillips

Phillips Associates

34137 N. Wooded Glen Drive

Grayslake, Illinois 60030

(847) 231-6068

www.phillipsassociates.com

[email protected]

Documents

Phillips Associates Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Learn 11 Surprising Techniques for Obtaining Powerful