ROI of Learning - Amazon Web Services...ROI of Learning Toolkit Prepared by Kristi Casey Sanders, CMP, CMM, DES, HMCC, Director, MPI Academy, Meeting Professionals International @KristiCasey

ROI of Learning ToolkitPrepared by Kristi Casey Sanders, CMP, CMM, DES, HMCC, Director, MPI Academy, Meeting Professionals International

@KristiCasey #MPI [email protected]

3 Steps in Measurement of Learning

Learning Environment

Learning Process

Learning into Action

2Copyright 2017, Meeting Professionals International, mpiweb.org

Learning Environment Components

If you can’t evaluate all, at least pick two.

DesignSetting

StructureLearner Readiness

Follow-Up


Design Score: ___ out of 90Organizers: Score 1-10 (10 being most true)

Learning offers appropriate level of challenge to participantsThere are activities to suit a wide

variety of learner stylesTechnology is used to aid interaction

with expertsThere is a mix of new and repeat

participantsThe content challenges the status quo

Participants are likely to encounter different points of viewThere will be opportunities to benefit

from collective as well as expert knowledgeParticipants will be able to put and test

learning in their own contextThe meeting encourages mixing

between more industry experienced and less experienced participants


Setting Score: ____ out of 70Organizers: Score 1-10 (10 being most true)

There are varied room layouts (theater-style, cabaret, boardroom, informal, etc.)

There is natural light

Refreshments are available during sessions

There will be spaces for participants to reflect on their own

There will be spaces for pairs to discuss learning

There are small group areas

The meeting provides a safe environment for people to participate


Structure Score: ____ out of 30Organizers: Score 1-10 (10 being most true)

There are frequent breaks

There is time for reflection at the end of sessions

Session timing is right for activities


Learner Readiness Score: ____ out of 60Organizers: Score 1-10 (10 being most true)

Speakers will show interest in the audience

Participant needs have been significantly used to inform the selection of content

The content will be very relevant to participants

Participants will receive learning materials in advance of the meeting

Participants will use an app for pre-scheduling and gaining information in advance of sessions

Participants come ready to learn


Follow-Up Score: ____ out of 70Organizers: Score 1-10 (10 being most true)

The learning will provide new insights

Social media will help to engage participants in the content

Sessions will be recorded for participants to use at the event and after the event on any device

Participants will be able to act upon their learning

Participants will be encouraged to contact other participants after the event

Participants are likely to access materials, handouts, slides after the event

This meeting is part of a year-round engagement strategy


Design Score: ____ out of 90Participants: Score 1-10 (10 being most true)

Learning offered the appropriate level of challengeThere were activities to suit my learner

preferenceI used technology to interact with

expertsI met a mix of new and repeat

participantsThe content challenged the status quo

I encountered different points of view

The meeting encouraged mixing between more industry experienced and less experienced participants

I felt I benefitted from collective as well as expert knowledge

I could put and test learning in my own context


Setting Score: ___ out of 70Participants: Score 1-10 (10 being most true)

There were varied room layouts (theater-style, cabaret, boardroom, informal, etc.)

There was natural light

Refreshments were available during sessions

There were spaces for me to reflect on my own

There were spaces for pairs to discuss learning

There were small group areas

This event provided a safe environment for me to participate


Structure Score: ____ out of 30Participants: Score 1-10 (10 being most true)

There were frequent breaks

There was time for reflection at the end of sessions

Session timing was right for activities


Learner Readiness Score: ____ out of 60Participants: Score 1-10 (10 being most true)

Speakers showed interest in the audience

I felt participant needs have been significantly used to inform the selection of content

The content was very relevant to me

I received learning materials in advance of the meeting

I used the event app for pre-scheduling and gaining information in advance of sessions

I came ready to learn


Follow-Up Score: ____ out of 70Participants: Score 1-10 (10 being most true)

The learning will provide new insights

I felt engaged by social media communications before, during and after the event

I intend to watch recordings of sessions filmed during the event

I will be able to use what I learned

I will stay in touch with people I met during the event

I intend to refer to session materials, handouts, slides after I return to work

I feel connected to this event’s community all year long


Assessment of Learning Environment Score Card

OrganizerElement Score

Design ____ /90 = _____

Setting ____ /70 = _____

Structure ____ /30 = _____

Learner Readiness ____ /60 = _____

Follow-Up ____ /70 = _____

TOTAL ____ /320 = ____

ParticipantElement Score

Design ____ /90 = _____

Setting ____ /70 = _____

Structure ____ /30 = _____

Learner Readiness ____ /60 = _____

Follow-Up ____ /70 = _____

TOTAL ____ /320 = ____

Average Learning Environment Score (organizer & participant) = _____


Learning Environment Observation Sheet To determine effectiveness of live learning environment, select a random sample of 10% of your sessions and have reviewers audit and rank using the following spectrum. Average

all session scores to determine average onsite score of the live learning environment.

10 123456789

What would your overall score for this session be? Rank the following along the above continuum of 1 to 10.

StructuredWell-TimedInteractiveHands-OnSpeaker Interest in AudienceOpenness to QuestionsAudience-Focused

UnstructuredRushedPassiveTheory

No InterestScripted

Distracted

SESSION SCORE: _____ /70 = _____

Also include reviewer name & notes; date, time & name of session; # of participants & description; session type; and room layout.


Learning Process Pre-Event Survey(feel free to customize/edit to meet your needs)

Meeting my peersMaking new contactsLearning new thingsPromoting my businessMeeting expertsLearning how to do my job betterGaining professional education creditsHaving a break from work

Seeing the destination

Feeling part of this community

Looking for new business

Looking for a new job

Strengthening my network

No specific objectives

Other (please specify) _____________

Participant instructions: Please select your top 3 objectives for attending

High-learning focused people will have 2-3 education-related goals; mid-learning focused will have 1; low-learning will have none. Use data to customize experience & pre-event communications.


Learning Process & Learning to Action Onsite Questionnaire

What learning has taken place?

What questions arise as a result of attending this session/activity?

What did you want to achieve at this meeting/training?

How would you describe your learning experience?

What did you do in advance of the meeting to prepare?

What do you think will happen to the learning after the conference?

Have staff or volunteers stop 1 in every 5 or 10 people at various points exiting sessions or sitting and meeting in specific spaces and ask if they could spare 5 minutes to answer a few questions. Interviewers should introduce themselves as a person acting on your behalf to find out about the experience. Interviews can be verbal, written or facilitated by technology in social, virtual,mobile or blended environments.

Questions for Participants


Learning Process & Learning to Action Onsite Questionnaire Scoring Key

Learning Process (score 0 to 10)• If no responses collected (sheet handed in blank,

nothing to say about learning) = 0• Acknowledgement (repeating info, describing

content) = 2.5• Contemplation (raising questions, using words like

I think, I believe) = 5• Reaction (showing usually emotional responses to

content) = 7.5• Leap to a new idea (showing connection, we can

use this where I work, or what about this) = 10

Learning to Action (score 0 to 10)• No response (no mention of action) = 0

• Application (I will use this, I am going to implement it) = 10

• Adaptation (I want to share this with colleagues, could use this) = 5

Average Learning Process Score = ____

Average Learning to Action Score = ____


Learning Environment, Learning Process and Learning to Action Data

Collection Method Average Score (out of 10)

Learning Environment Evaluations by Organizers and Participants

Learning Environment Observations Onsite

Learning Process Pre-Event/Post-Event Survey

Learning Process & Learning to Action Onsite Questionnaire

Technology (app polls, website & social media behavior)

Traditional Survey Data

TOTAL AVERAGE LEARNING SCORE


Calculating the ROI of Learning


Determine Your Average Learning Score From All Evaluations, Data, Surveys &

Observations• Average Learning Environment Score = ____ out of 10

• Average Learning Process Score = ____ out of 10

• Average Learning into Action Score = ____ out of 10

The average of all three = Average Learning Score = _______ out of 10

Learning Environment

Learning Process

Learning into Action


Determine the Average Learning Cost Per Hour

Total $ spent on average could be:• How much this training costs your company to

produce. • Fully loaded costs could include lost productivity

while learners are out of the office (multiply average per-person hourly salary by hours spent out of office) & how much attendees spend on travel/hotel

• Or, what the competition charges for similar conferences

• Or, what other trainings like this cost in your region

• For free trainings, what is the cost of lost productivity per hour and attendee travel expenses?

• A = Total $ spent, on avg. = $______• B = Total # of learning hours = ___• A / B = Average cost per training hour• Average cost per training hour = $______• Total # of educational hours you’re offering

= ___________• Average per-person registration rate

= $____________

Now calculate the per hour cost:


Calculate the ROI of learning

What was the average learning score? (ex., 7.5)

What is that divided by 10? (ex., .75 or 75%)

What is the avg. cost per training hour times the number of

educational hours you’re offering?

What is #2 multiplied by #3?

What is the average per-person registration fee for your event?

Divide #4 by #5 and multiply by 100 to get ROI of learning%.


The ROI of Learning Formula

Average Learning Score

10

Average Cost Per Training Hour

Total Number of Training or Education

Hours in Your Program

Average Registration Cost Per Person



6.5

10$125 21

$1,200

$2,625.65

$1,200

1.4 x 100140% ROI



________

10

$__________ ___________

$ _____________

= _____ Benefit to Cost Ratio or _____% ROICopyright 2017, Meeting Professionals International, mpiweb.org 26

Visit mpiweb.org for more education & inspiration

ROI of Learning Toolkit was prepared by Kristi Casey Sanders, CMP, CMM, DES, HMCC, Director, MPI Academy from Meeting Professionals

International’s research on Measuring Meeting Value and its Measuring Meeting Value Guidebook. More education relating to proving the value of meetings, unique learning formats, event production and experience

design may be found at academy.mpiweb.org.

Documents

ROI of Learning - Amazon Web Services...ROI of Learning Toolkit Prepared by Kristi Casey Sanders, CMP, CMM, DES, HMCC, Director, MPI Academy, Meeting Professionals International @KristiCasey