Phillips Associates 1

Presented by: Ken Phillips Phillips Associates September 11, 2012

DEMONSTRATE VALUE AND GET

YOUR “SEAT AT THE TABLE”

51 TIPS & BEST KEPT SECRETS FOR

DESIGNING LEVEL 1 - 4 EVALUATIONS

Phillips Associates 2

AGENDA

➤ Measurement and evaluation essentials ➤ Create Level 1 evaluations that obtain

powerful data ➤ Crack the code of test question design

and take your Level 2 evaluations to 2.5 ➤ Capture elusive Level 3 data: the secrets

of survey design ➤ Level 4 evaluations ➤ Making sense out of evaluation data

Phillips Associates 3

M & E OVERARCHING GOALS

Decide whether to continue or discontinue a learning program

Improve a learning program

Demonstrate value of a learning program to executives/stakeholders

Improve management of learning function – run learning like a business

Phillips Associates 4

Aggregating data collected about a learning

program, interpreting and drawing conclusions

from the results and making recommendations

based on the conclusions

MEASUREMENT vs. EVALUATION

Collecting data about the outcome of a learning program

Measurement =

Evaluation =

Phillips Associates 5

Levels of Evaluation Measurement Focus Time Frame

Level 1: Reaction Participant reaction to a learning program

Conclusion of learning program

Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes

Conclusion of learning program or within 6 to 8 weeks after

Level 3: Behavior Degree to which participants applied back-on-the-job what was learned

2 to 12 months

Level 4: Results Degree to which targeted business outcomes were achieved

9 to 18 months

Level 5: ROI Degree to which monetary program benefits exceed program costs

9 to 18 months

KIRKPATRICK/PHILLIPS EVALUATION MODEL

Phillips Associates 6

LEVEL OF EVALUATION BEING MEASURED?

➤ Situation 1: Participants will be able to

distinguish good examples of

paraphrases from poor examples.

Level 2: Knowledge

Phillips Associates 7

LEVEL OF EVALUATION BEING MEASURED?

➤ Situation 2: Participants provide

feedback on the relevancy of a learning

program to them and their job.

Level 1: Reaction

Phillips Associates 8

LEVEL OF EVALUATION BEING MEASURED?

➤ Situation 3: Six months after a learning

program employee engagement scores

increase by 10% over the previous year.

Level 4: Results

Phillips Associates 9

LEVEL OF EVALUATION BEING MEASURED?

➤ Situation 4: The estimated benefit of the

increase in employee engagement is

$250K and the program costs were

$150K.

Level 5: ROI

Phillips Associates 10

LEVEL OF EVALUATION BEING MEASURED?

➤ Situation 5: One week following a

learning program customer service rep

follow-up phone calls increase by an

average of 6 per rep.

Level 3: Behavior

Phillips Associates 11

“CHAIN OF EVIDENCE” DETERMINENTS

Strategic importance of learning program

Cost of learning program

Number of participants attending program

Stakeholder/business executive expectations

Phillips Associates 12

11 SURPRISING TECHNIQUES FOR

OBTAINING POWERFUL DATA FROM

LEVEL 1 EVALUATIONS

Phillips Associates 13

LEVEL 1 FACTS*

92% of organizations evaluate at least some learning programs at Level 1

Organizations that use Level 1s on average evaluate 78% of all learning programs

36% of organizations view data collected as having high or very high value

*2009 ASTD Research Study: The Value of Evaluation: Making Training Evaluations More Effective

Phillips Associates 14

WHY THE DISCONNECT?

➤ Level 1 Evaluations focus on learning department not business operation data

➤ Examples • Facility • Course design • Facilitator • Food

➤ The data collected is not used

Phillips Associates 15

WHAT’S THE SOLUTION?

➤Create Level 1 Evaluations that produce data with high perceived value for both you & your business stakeholders

➤Aggregate data and conduct trend lines or comparisons at group, project, department, regional level

Phillips Associates 16

*Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006.

DATA COLLECTION METHODS*

We hope you have found this excerpt interesting. To get access to the full version, please consider becoming a member of the Northeast Wisconsin Chapter of ASTD.

Go to

http://www.new.astd.org/Default.aspx?pageId=955638

to join, today!