Sharing Lessons on Learning Metrics

Sharing Lessons on Learning Metrics

LINGOs Annual Meeting Sept. 23-24, 2009Steven Honeyman and Marie-Laure CurieGlobal Capacity Building Department

Let’s start with small group discussions• Briefly describe your learning metrics (overview only)• Discuss advantages and disadvantages of different

components • Summarize what the group thinks are some new or

interesting ideas the whole group should know about

LINGOs Annual Meeting - Sept. 23-24, 2009

Kirkpatrick’s model

Includes four levels/steps of outcome evaluation:• Level 1—Reaction – satisfaction survey• Level 2—Learning – tests • Level 3—Behavior – on-the job behavior change

survey• Level 4—Results – business outcomes


Measuring Training Impact at PSI

SMILE - Social Marketing Initiative for Learning and Excellence

Introduction to the PSI Behavior Change Framework

• Behavior change framework drives PSI marketing activities– Fundamental to social marketing

• Applicable to all behaviors promoted by PSI• Based on behavior change theory• Determinants measured through TRaC (Tracking

Results Continuously) survey


PRODUCT PRICE PLACE PROMOTION

SOCIAL MARKETING INTERVENTION

OPPORTUNITY ABILITY MOTIVATION

POPULATION CHARACTERISTICS

AT RISK POPULATIONS

RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE

IMPROVED HEALTH STATUS/QUALITY OF LIFE

Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

The overall framework






AT RISK POPULATIONS



Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

What PSI does…

Our sphere of influence






AT RISK POPULATIONS



Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

What we try to influence…






AT RISK POPULATIONS



Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

To change a behavior...






AT RISK POPULATIONS



Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

That improves health


OAM

• Opportunity: institutional or structural factors that influence an individual’s chance to perform a promoted behavior.

• Ability: an individual’s skills or proficiencies needed to perform a promoted behavior.

• Motivation: an individual’s desire to perform a promoted behavior.


• What is it?– Repetitive quantitative survey that measures behavior change

of a specific behavior of a specific target group– Measure the effectiveness of project interventions and

activities• Why do we use it?

– To better design programs, to monitor program progress (ensure that the target population is adopting the promoted behavior and if needed, make adjustments) and evaluate how we did (target group and behavior)

– Informs decision-maker where to invest more/fewer resources• Who does it?

– All PSI programs • When does it occur? - Prior, during (sometimes often) and at the end of an

intervention

TRaC (Tracking Results Continuously))


CAPACITY BUILDING INTERVENTION



BY PSI STAFF

USE OF SKILLS ACQUIRED AT THE TRAINING

IMPROVED JOB PERFORMANCE AND OVERALL IMPACT OF PSI PROGRAMS

Availability

Brand Attributes

Locus of control

Willingness to pay

Outcome expectation

Belief

Threat

Attitudes

Intentions

Subjective norm

Self Efficacy

Social Support

Knowledge

Social Norm

Quality of Care

Brand Appeal

The overall Capacity Building framework


• What is it?– Short, quantitative survey– Measures application of training on-the-job

• Why do we use it?– Programmatic check to ensure that the training had the

intended impact of transferring needed skills/behaviors to staff and to identify gaps in training content/delivery

– Informs CB decision-maker where to invest more/fewer resources

• Who does it?– Global Capacity Building department (for virtual training as

well)• When does it occur?

– Approx. 6 to 8 weeks after the training ended

TRaC-T (Tracking Results Continuously for Training))


Kirkpatrick’s model

Includes four levels/steps of outcome evaluation:• Level 1—Reaction – satisfaction survey• Level 2—Learning – tests • Level 3—Behavior – TRaC-T survey• Level 4—Results – business outcomes


TRaC-T survey (example)



Some TRaC-T results

TRaC-T Budget Review training - August 5 & 6 2009



Some TRaC-T results

TRaC-T might have some limitations:• Self reported behavior, • Needs to be built into the training program,• Needs resources to develop and manage, • May not apply learning right away.

But it has many potential benefits:• Demonstrates training effectiveness to senior managers, • Convinces supervisor of the value of a particular training, • Allows targeting of training resources to highly effective training,• Provides feedback to those leading training on whether they are

being effective, and• Can be automatically set-up for online training !


For more information:

• www.psi.org/research/• Marie-Laure Curie, Manager, GCB Dept.,

[email protected]• Steven Honeyman, Director, GCB Dept.

[email protected]


Documents

Sharing Lessons on Learning Metrics