1 Show Me The Money Measuring ROI for People, Projects, and Programs Jack J. Phillips, Ph.D

1

Show Me The Money

Measuring ROI for People, Projects, and Programs

Jack J. Phillips, Ph.D.

Learning Objectives

• Make the business case for "showing the money" for a few select programs

• Identify the steps, guiding principles, and issues involved in "showing the money"

• Use impact and ROI data to drive improvement and change

2

3

The Value EvolutionThe Value Evolution

The Many Applications of the ROI Methodology

Communications Policies/Procedures

Consulting/OD Procurement/Supply Chain

Educational Systems Public Policy Programs

Human Resources/Human Capital

Public Relations / Public Affairs

Leadership and Coaching Quality / Six Sigma

Learning and Development Research and Development

Marketing/Advertising Social Programs

Meetings and Events Technology / Systems

4

The “New” Definition of Value

Value Must:• Be balanced, with qualitative and quantitative data• Contain financial and non-financial perspectives• Reflect strategic and tactical issues• Represent different time frames• Satisfy all key stakeholders• Be consistent in collection and analysis• Be grounded in conservative standards• Come from credible sources• Reflect efficiency in its development• Create a call for action

5

The Changing Face of Value Old New

Activity cost center Results profit center

Expense control Maximize Value

Human Resources as expenses Human capital as investment

Rule centered Client centered

We value what IBM and GE value Our value systems are unique

Tolerate overhead Outsource or automate overhead

Add value with small pieces Add value with integrated applications

Most of us are in a support role All of us are in a sales role

Our measures are based on benchmarking

Our measures are based on what we need

We view value from one perspective We view value from everyone’s perspective

Just another days work in the cubicle We do something of value

A job A performance

Treat old ideas as new ideas Treat old ideas as old ideas

6

The “Show Me” EvolutionTerm Issue

Show Me! Collect Impact Data

Show Me the Money! And Convert Data to Money

Show Me the Real Money!And Isolate the Effects of the

Project

Show Me the Real Money, And Make Me Believe it!

And Compare the Money to the Cost of the Project

7

How “Show Me the Money” Connects

8

SHOW METHE MONEY!!

Organizational Measures(Sponsor)

Personal Measures

(Stakeholders)

FinancialMeasures(CFO)

Varieties of Terms and Applications

Name Example

Program A leadership development process for senior executives

Project A re-engineering project for the plastics division

System A fully networked system for all branches

Initiative A faith-based initiative to reduce recidivism

Policy A new pre-school policy for the disadvantaged

Procedure A new scheduling procedure for truck drivers

Event A golf outing for customers

Meeting US Coast Guard innovations conference

Process Quality sampling process

People Staff additions in the customer care center

Tool A new selection tool for the hotel staff9

Results• reacted very positively to the program and found it to be

very relevant to their work;• learned new skills and gained new insights about

themselves;• utilized the skills and insights routinely with their teams,

although they had some difficulty in a few areas;• improved several important work unit measures, with

some measures improving as much as 28%;• achieved an impressive 105% return on investment; and• reported an increase in job satisfaction in the work unit.

10

Key Issues with this Key Issues with this Level of AnalysisLevel of Analysis

• Objectives?• Credibility of data?• Source of data?• Consistent methodology?• Scope?• Standards?• Use of data?• Cost of process?• Fear of data?

11

Project or Program Value ChainProject or Program Value ChainLevel Measurement Focus

0. Input Measures input such as volume and efficiencies

1. Reaction & Planned Action

Measures participant reaction to the program and captures planned actions

2. Learning Measures changes in knowledge, skills, and attitudes

3. Application Measures changes in on-the-job behavior or actions and progress with actions

4. Business Impact Captures changes in business impact measures

5. ROI Compares program benefits to the costs

Global Trends in Global Trends in Measurement and Evaluation Measurement and Evaluation

• Organizations are moving up the value chain in their evaluation strategies

• Investment is increasing to 3-5% of the budget

• Increase focus is driven by clients and sponsors

• ROI is the fastest growing metric

13

. . . and

Global Trends in Global Trends in Measurement and Evaluation Measurement and Evaluation

• Evaluation data is used to drive improvement and secure funding

• Evaluation is addressed early and often in the implementation cycle

• Processes are systematic and methodical, often designed into the delivery and implementation processes

• Technology is significantly enhancing processing

14

15

Activity Based Results Based

• No business need for the program • Program linked to specific business

• No assessment of performance issues

• Assessment of performance effectiveness

• No specific measurable objectives • Specific objectives for behavior & business impact

• No effort to prepare program participants to achieve results

• Results expectations communicated to participants

Paradigms are shifting to results?Paradigms are shifting to results?

and . . .

16

Activity Based Results Based

• No effort to prepare the work environment to support transfer

• Environment prepared to support transfer

• No efforts to build partnerships with key managers

• Partnerships established with key managers and clients

• No measurement of results or benefit-cost analysis

• Measurement of results and benefits-cost analysis

• Planning and reporting is input focused

• Planning and reporting is outcome focused

Paradigms are shifting to results?Paradigms are shifting to results?

17

Why does ROI work?

• Provides a balanced set of measures• Offers a step-by-step process• Bridges business evaluation and program

evaluation• Balances research/statistical methods with

practical application• Flexible for all types of programs• Credible with managers and administrators

18

Reliance Insurance

19

Is Your Organization a Candidate for ROI Implementation?

20

Scoring

• If you scored:

15 – 30 You are not yet a candidate for ROI.

31 – 45 You are not a strong candidate for ROI, however, it is time to start pursuing some type of measurement process.

46 – 60 You are a candidate for building skills to implement the ROI process. At this point there is no real pressure to show the ROI, which is the perfect opportunity to perfect the process within the organization.

61 – 75 You should already be implementing a comprehensive measurement and evaluation process, including ROI.

21

Private Sector Organizations using ROI Include:

• Accenture• Allstate Insurance• Apple Computer• AT&T• Bank of America• Banner Healthcare• Baptist Medical

Center• Bristol-Myers Squibb• Children’s Hospital

of Los Angeles• Caremark RX• Covenant Healthcare

Systems• Dell Computers

• Deloitte & Touche• Delta Airlines• DHL Worldwide

Express• Federal Express• Genentech• Georgia Pacific• Guthrie Healthcare

Systems• Hewlett-Packard• Intel• Lockheed Martin• Memorial Foundation

Hospital

• Menlo Logistics• Microsoft• Motorola• NCR• Nextel• Pricewaterhouse

Coopers• QUALCOMM• Shell Oil• UPS• Wachovia Bank• Wal-Mart

22

Public Sector Organizations using ROI Include:

• US Department of Defense

• US Department of Navy

• US Department of Labor

• US National Security Agency

• Central Intelligence Agency

• US Department of Veteran’s Affairs

• NASA

• State of Mississippi

• State of Texas

• State of New York

• Government of New Zealand

• Government of Singapore

• Government of Italy

• Government of Australia

• Government of Canada

23

Generates six types of measures:

• Reaction and Planned Actions

• Learning

• Application and Implementation

• Business Impact

• Return on Investment

• Intangible Measures

. . . and includes a technique to isolate the effects of the program or solution

The ROI Methodology

24

• 5,000 impact studies each year.

• 20,000 individuals have attended a two-day ROI workshop.

• 4,000 individuals are certified to implement the ROI methodology.

• The ROI methodology has been adopted by hundreds of organizations in manufacturing, service, non-profit, and government settings in 44 countries.

The ROI Methodology is Grounded in Research and Application

and . . .

25

• The process has been refined over a 20-year period.

• 16 books have been developed to support the process.

• ROI Network has 600 members

• Several ROI Network conferences are conducted annually

• Ongoing research on the ROI methodology is conducted by the ROI Institute, Inc. to support practitioners.

The ROI Methodology is Grounded in Research and Application

26

The ROI Calculation

Program Benefits

Program Costs

Net Program Benefits

Program Costs

BCR =

ROI = X 100

Measurement of Communication ProjectsLevel Measurement

CategoryCurrent Status

Coverage

Recommending Comments About Status

0 Inputs/IndicatorsMeasures inputs into communication projects including number of projects, audience, costs, and efficiencies.

100% 100%This is being accomplished now.

1 Reaction and Perceived ValueMeasure reaction to, and satisfaction with, the medium, content, and value of the communication.

80-100%

Need more focus on content and perceived value.

2 LearningMeasures what participants understand or learned from the communication – information, knowledge, skills, and contacts (take-aways).

50-60%Must use simple learning measures.

27

28

3 Application and ImplementationMeasures progress after the communication – the use of information, knowledge, skills, and contacts.

15-25%Need more follow-up.

4 Impact and ConsequencesCaptures changes in business impact measures such as output, quality, time, and cost-linked to the communication.

10%

This is the connection to business impact.

5 ROICompares the monetary benefits of the business impact measures to the costs of the project.

5%The ultimate evaluation.

29

Five Levels of Measurement - Examples

• Level 0 Input and Indicators

• Level 1 Reaction and Perceived Value

• Level 2 Learning

• Level 3 Application and Implementation

• Level 4 Impact and Consequences

• Level 5 Return on Investment

Five Levels of Measurement - Examples

30

31

Evaluation is like a Puzzle

An An EvaluationEvaluationFrameworkFramework

Case Case ApplicationsApplicationsand Practiceand Practice

A A ProcessProcessModelModel

OperatingOperatingStandards Standards

andandPhilosophyPhilosophy

Implementation

32

ROI Methodology

Develop EvaluationPlans and

Baseline Data

Develop EvaluationPlans and

Baseline Data

Stage 1Evaluation Planning

Stage 2 Data Collection

Collect DataDuring SolutionImplementation

Collect DataDuring SolutionImplementation

Collect DataAfter Solution

Implementation

Collect DataAfter Solution

Implementation

DevelopObjectivesOf Solution

DevelopObjectivesOf Solution

Level 1

Level 2

Level 3

Level 4

33

CaptureCosts

Of Solution

CaptureCosts

Of Solution

Isolate theEffects ofSolution

Isolate theEffects ofSolution

Convert Data to Monetary

Value

Convert Data to Monetary

Value

Calculatethe Return On

Investment

Calculatethe Return On

Investment

Generate Impact Study

Report

Generate Impact Study

Report

Stage 3 Data Analysis

Identify IntangiblesMeasures

Identify IntangiblesMeasures

Stage 4 Communicate

Results

Level 5

Intangible Benefits

34

1. When a higher level evaluation is conducted, data must be collected at lower levels.

2. When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive.

3. When collecting and analyzing data, use only the most credible sources.

4. When analyzing data, choose the most conservative among alternatives.

5. At least one method must be used to isolate the effects of the project/initiative.

6. If no improvement data are available, it is assumed that little or not improvement has occurred.

The Guiding Principles

and . . .

35

7. Estimates of improvement should be adjusted for the potential error of the estimate.

8. Extreme data items and unsupported claims should not be used in ROI calculations.

9. Only the first year of benefits (annual) should be used in the ROI analysis of short-term projects/initiatives.

10.Project/program costs should be fully loaded for ROI analysis.

11.Intangible measures are defined as measures that are purposely not converted to monetary value.

12.The results from the ROI methodology must be communicated to all key stakeholders.

The Guiding Principles

36

Reaction & Planned Action

Learning

Application & Implementation

Impact

ROI

Isolate the Effects of the Program

Report the Chain of Impact

Intangible Benefits

37

Reaction & Planned Action

Learning

Application & Implementation

Organization Impact

Stakeholder 2 Impact

Stakeholder 3 Impact

ROIIntangibles

Multiple Stakeholder Chain of Impact

ROIIntangibles

ROIIntangibles

38

Characteristics of Evaluation Levels

Chain of Value of Difficulty ofImpact Information of Use Assessment

Lowest Frequent EasyReaction

Learning

Application

Impact

ROI Highest Infrequent Difficult

Consumers = The customers who are actively involved in the process.

Client = The customers who fund, support, and approve the project

Focus

Consumer

Client

Power to Show

Results

Highest

Lowest

39

Changes in key outcome measures

Changes in key outcome measures

Cost-benefit comparison

Cost-benefit comparison

Changes inperformance

Changes inperformance

Need for skills orknowledge

Need for skills orknowledge

PreferencesPreferences

Level 5Level 5

Level 4Level 4

Level 3Level 3

Level 2Level 2

Level 1Level 1

© 2001 Patricia Pulliam Phillips

Evaluation Purpose

Program Need

Program Profile

Stakeholder Needs

Evaluation Purpose

Program Need

Program Profile

Stakeholder Needs

40

When Selecting Programs for Level 4 and 5 Evaluation, Consider the Following:

• Life cycle of the program• Linkage of program to operational goals and issues• Importance of program to strategic objectives• Top administrator interest in the evaluation• Cost of the program• Visibility of the program• Size of target audience• Investment of time required

41

Evaluation TargetsPercent of Courses

Level Current Target Suggested

Reaction

Learning

Application (Behavior)

Impact

Return on Investment

90 - 100%

40 – 60%

30%

10 - 20%

5 - 10%

42

• Executive Education• Leadership• Diversity Programs• Wellness/Fitness Initiatives• Total Quality Management• Self-Directed Teams• Skill-Based/Knowledge-

Based Compensation• Organizational

Development• Meeting Planning• Competency Systems

• Career Development Programs

• Recruiting Strategies• Orientation Systems• Associate Relations

Programs• Gainsharing Programs• Technology Implementation• e-Learning• Safety & Health Programs• Project Management

ROI Application

43

1. Reaction

2. Learning

3. Application

4. Business Impact

5. Return on Investment

Matching Evaluation Levels with Objectives

44

Program Objectives Provide

• Direction to designers and developers

• Guidance to instructors and facilitators

• Goals for participants

• Satisfaction for program sponsors

• A framework for evaluators

45

Needs ProgramAssessment Objectives Evaluation

Business Impact BusinessNeeds Objectives Impact

Job Performance Application ApplicationNeeds Objectives

Skills/Knowledge Learning LearningNeeds Objectives

Preference Satisfaction ReactionNeeds Objectives

4 4

3 3

2 2

1 1

Payoff ROI ROINeeds Objectives5 5

46

Absenteeism is costing $100,000 month

Unexpected absenteeism problem exists

Dialogue between team leader/supervisor is not

occurring when there is an unexpected absence

Deficiency in counseling/discussion skills

Two-day communication skills workshop must provide

usable and relevant skills; facilitator-led; participants

include supervisors and team members

5

4

3

2

1

ROI of at least 25%

Weekly absenteeism rate will reduce by 10% six

months after course

Dialogue takes place in 95% of situations when

an unexpected absence occurs. Environment

supports safe dialogue.

Acquisition of dialogue skills are demonstrated

Program receives favorable rating of 4 out of 5 on

relevance; participants identify three planned

actions

Calculate the ROI

Monitor absenteeism data for six months

Follow-up questionnaire to participants to check frequency

of skill application Identify barriers to dialogue- three

months post program

Skill practice sessions during program

Reaction questionnaire at the end of program

5

4

3

2

1

47

Key Alignment QuestionsKey Alignment Questions

Potential ROI ROIPayoffs Objectives5

5

Is this a problem worth solving?

Is there a potential pay off?

What is the actual ROI?

What is the BCR?


48


44

What is the specific measure?

What happens if we do nothing?

Which business measure improved?

How much is related to the program?


Business Impact BusinessNeeds Objectives Impact

49


33

What is occurring or not occurring on the job that influences the business measure?

What has changed?

Which skills/knowledge has been applied?


Job Performance Application ApplicationNeeds Objectives

50


22

What skills or knowledge is needed to support the job performance need?

What did they learn?

Who did they meet?


Skills/Knowledge Learning LearningNeeds Objectives

51


11

How should the solution be structured?

What was the reaction to the program?

Do we intend to implement the program?


Satisfaction Preferences Objectives Reaction

52

Developing Developing Reaction ObjectivesReaction Objectives

Developing Developing Reaction ObjectivesReaction Objectives

Developing Developing Learning ObjectivesLearning Objectives

Developing Developing Learning ObjectivesLearning Objectives

53

Developing Application Developing Application ObjectivesObjectives

Developing Application Developing Application ObjectivesObjectives

Developing Impact Developing Impact ObjectivesObjectives

Developing Impact Developing Impact ObjectivesObjectives

54

Developing Level 3 Developing Level 3 and 4 Objectivesand 4 Objectives

Developing Level 3 Developing Level 3 and 4 Objectivesand 4 Objectives

55

Option 1, When You Don’t Have a Clue

Option 2, When the Measure is in a Defined Set

Option 3, When the Measure is Known

56

Plan Your Project Evaluation Worksheet

57

Data Collection Plan

58

ROI Analysis Plan

59

•Follow-Up Surveys•Follow-Up Questionnaires•Observation On the Job•Interviews with Participants•Follow-Up Focus Groups•Program Assignments•Action Planning•Performance Contracting•Project Follow-Up Session•Performance Monitoring

Level 3

Level 4

Collecting Post Program Data

60

• Type of data

• Time – Participant / Supervisor

• Costs

• Accuracy – Validity / Reliability

• Utility

• Culture / Philosophy

Factors to Consider When Selecting Data Collection Methods Include:

61

Factors to Consider When Determining Timing of Follow-up Include:

• Availability of data

• Ideal time for behavior change (Level 3)

• Ideal time for business impact (Level 4)

• Convenience of collection

• Constraints on collection

62

• Use of a control group arrangement

• Trend line analysis of performance data

• Use of forecasting methods of performance data

• Participant’s estimate of program impact (percent)

• Supervisor’s estimate of program impact (percent)

• Manager’s estimate of program impact

• Use of expert/previous studies

• Calculate/estimate the impact of other factors

• Customer input

Methods to Isolate Program Effects

Applications of Data Collection Instruments

A – Survey

B – Test

C – Questionnaire

D – Interview

E – Focus Group

F – Observation

G – Performance Records

63

64

International SalesInternational Sales

65

Control Group Method Design

ExperimentalGroup M1 Program M2

Control Group M1 M2

66


Control Group

ExperimentalGroup M1

M1

Program

67


Experimental

Control

Program

Experimental(No pre-measure)

M1 M2

M1 M2

Program M1

68

2%

1%

J F M A M J J A S O N D J

ERRORRATE

1.85% Pre Program Average

1.45% Projected Average

MONTHS

Example of Trend Line AnalysisExample of Trend Line Analysis

CPI Program Conducted

.7% Post Program Average

Isolating the Effects of a Program

A – Control Group

B – Trend line analysis

C – Forecasting

D – Participant’s estimate

E – Use of customer input

F – Expert’s estimate

69

70

Factor that Influenced

Improvement

Percent of Improvement

Caused By

Confidence Expressed as a

Percent

Adjusted Percent of

Improvement Caused By

Training Program

60% 80% 48%

System Changes 15% 70% 10.5%

Environmental Changes

5% 60% 3%

Compensation Changes

20% 80% 16%

Other ___% ___% ___%

Total 100%

Example of Estimation

71

Credibility of Data is Influenced by:• Reputation of the source

Source of data Source of the study

• Biases Motives of the researcher Personal bias of the audience

• Methodology Assumptions made in the analysis Realism of the outcome data Type of data

• Scope of analysis

72

Data are Converted by:

• Converting output to contribution – standard value• Converting the cost of quality – standard value• Converting employee’s time – standard value• Using historical costs• Using internal and external experts• Using data from external databases• Linking with other measures• Using participants’ estimates• Using supervisors’ and managers’ estimates• Using staff estimates

73

Example: Cost of One Turnover from External Database

Salary of Middle Manager $70,000/annually

Value of Turnover* 150% of annual salary

Cost of Turnover $105,000

* Value obtained from industry-related study (external data)

74

Cost of a Sexual Cost of a Sexual Harassment ComplaintHarassment Complaint

35 Complaints35 Complaints

Actual Costs from RecordsActual Costs from Records

Additional Estimated Costs from Staff

Additional Estimated Costs from Staff

Legal Fees, Settlements, Losses, Material, Direct

Expenses

Legal Fees, Settlements, Losses, Material, Direct

Expenses

EEO/AA Staff Time, Management Time

EEO/AA Staff Time, Management Time

$852,000 Annually$852,000 Annually

Cost per complaint = $24,343Cost per complaint = $24,343$852,000

35

75

To Convert or Not Convert

• Is there a standard value?

• Is there a method to get there?

• Can we get there with minimum resources?

• Can we convince our executive in two minutes that the value is credible

Converting Data to MoneyA – Profit/savings form output

B – Standard Value

C – Employee time as compensation

D – Historical Costs/savings form records

E – Expert input

F – External database

G – Linking with other measures

H – Participant estimation

I – Management estimation

J – Estimation for HR staff76

77

5-Step Data Conversion1. Focus on a unit of measure

2. Determine the value (V) of each unit

3. Calculate the change in performance data (∆P)

4. Determine an annual amount for the change (A∆P)

5. Calculate the total value of the improvement (A∆P x V)

78

Example using Internal Experts

Step 1: One grievance

Step 2: V = $6,500 (from Director of Nursing and HR experts)

Step 3: ∆P = 7 out of 10 grievances prevented per month due to program

Step 4: Annual ∆P = 84

Step 5: A∆P x V = $546,000

79

A BOperating CostsSupport Costs

Administrative CostsParticipant Compensation and Facility CostsClassroom Costs

C D

Program Development CostsAdministrative CostsClassroom CostsParticipant Costs

Analysis CostsDevelopment CostsDelivery CostsEvaluation Costs

Which Cost Category is Appropriate for ROI?

80

• Assessment Costs (Prorated)

• Development Costs (Prorated)

• Program Materials

• Instructor/Facilitator Costs

• Facilities Costs

• Travel/Lodging/Meals

• Participant Salaries and Benefits

• Administrative/Overhead Costs

• Evaluation costs

Fully-Loaded Costs Profile

81

Potential Intangible Benefits

• Increased Job Satisfaction

• Increased Organizational Commitment

• Improved Teamwork

• Improved Customer Service

• Reduced Complaints

• Reduced Conflicts

• Reduced Stress

82

Program Benefits

Program Costs

Net Program Benefits

Program Costs

BCR =

ROI (%)= X 100

ROI is Reported One of Two Ways

83

Calculate the ROI

Costs per program (25 participants) $80,000

Benefits per program (1st year) $240,000

BCR = =

ROI = x 100 =

$240,000

$80,0003

$160,000

$80,000200%

84

What is an Acceptable ROI?

• Set the value as with other investments -15%

• Set slightly above other investments - 25%

• Set at break even - 0%

• Set at client expectations

ROI

Impact

Application

Learning

Reaction and Satisfaction

Level 5: 5-10%

Level 1: 90-100%

Level 2: 40-60%

Level 3: 30%

Level 4: 10-20%

85

Evaluation Targets

86

Criteria for Selecting Programs Criteria for Selecting Programs for Levels 4 & 5for Levels 4 & 5

• Expected life cycle of the program

• The importance of the program in meeting the organization’s goals

• Cost of the program

• Visibility of the program

• The size of the target audience

• Extent of management interest

87

When Properly Implemented, High ROI Values can be Achieved with Programs on:

• Leadership• Team Building• Management Development• Supervisor Training• Sales Training

. . . 100% to 700% ROI is not uncommon

88

ROI Best Practices

1. The ROI methodology is implemented as a process improvement tool and not a performance evaluation tool for the learning/development staff.

2. ROI impact studies are conducted very selectively, usually involving 5-10% of programs.

3. A variety of data collection methods are used in ROI analysis.

4. For specific ROI evaluations, the effects of learning/development are isolated from other influences.

5. Business impact data are converted to monetary values. and . . .

89

ROI Best Practices6. ROI evaluation targets are developed, showing the

percent of programs evaluated at each level.

7. The ROI methodology generates a micro level scorecard.

8. ROI methodology data are being integrated to create a macro scorecard for the learning/development function.

9. The ROI methodology is being implemented for about 3-5% of the learning/development budget.

10. ROI forecasting is being implemented routinely

11. The ROI methodology is used as a tool to strengthen/improve the learning/education process.

90

Time/Cost Savings Tips

1. Plan for Evaluation –Early

2. Build in Evaluation Tools

3. Share Responsibilities

4. Communicate Expectations

5. Use Standard Tools and Templates

91

Time/Cost Saving Tips

6. Use Estimates

7. Use Shortcuts Methods

8. Use Sampling, Routinely

9. Streamline Reporting

10. Develop Internal Capability

11. Use Technology

92

Healthcare, Inc.

Sexual Harassment Prevention Workshop

93

Why be Concerned with Communication?

• Measurement and evaluation are meaningless without communication

• Communication is necessary for making improvements

• Communication is a sensitive issue

• Different audiences need different information

94

Communication Principles

• Keep communication timely

• Target communication to specific audiences

• Carefully select communication media

• Keep communication consistent with past practices

• Incorporate testimonials from influential individuals

• Consider the training function’s reputation when developing the overall strategy

95

There are Four Types of Reports

1. Complete Report

2. Executive Summary

3. General Audience Summary

4. Streamlined Report

96

The Complete Report Includes the Details

• General Information• Methodology for Impact Study• Data Analysis• Costs• Results• Barriers and Enablers• Conclusions and Recommendations• Exhibits

97

Sample Table of Contents of an ROI Impact Study

Sample Table of Contents For an Executive Summary

98

Sprint/Nextel

• Program Title: Diversity

• Target Group: Managers and Employees

• Solution: All-Inclusive Workforce Program (AIW)

99

• Level 1: Reaction• Composite Rating: 4.39 out of 5 (for six

items)

• Level 2: Learning• Averaged 4.28 out of 5 (for learning on six

objectives)

Sprint/Nextel

100

• Level 3: Application Managers:• Supports AIW (87%)• Addresses Problems (81%)• Encourages Staff (78%)Employees:• Supports AIW (65%)• Identifies Differences (63%)• Encourages Staff (60%)

• 91% of Managers successful completed action plans

Sprint/Nextel

101

• Level 4: Impact• Attrition Rate Improvement = 9.77%

• Level 5: ROI • BCR: 2.6• ROI: 163%

• Intangible Benefits• Employee Satisfaction• Communication• Cooperation• Diversity Mix• Teamwork

Sprint/Nextel

102

Sprint/Nextel

• Technique to Isolate Effects of Program: Manager’s estimate, adjusted for error

• Technique to Convert Data to Monetary Value: Standard cost item ($89,000 per Turnover)

• Fully-loaded Program Costs: $1,216,836

103

Building an ROI Scorecard

• Provides macro-level perspective of success

• Serves as a brief report versus detailed study

• Shows connection of training’s contribution to business objectives

• Integrates various types of data

• Demonstrates alignment between programs, strategic objectives, and operating goals

104

The Scorecard Includes Seven Types of Data

Indicators / Scope / Volume

Level 1 – Reaction / Satisfaction

Level 2 – Learning

Level 3 – Application / Barriers / Enablers

Level 4 - Business Impact

Level 5 - ROI

Intangibles

105

Reporting for a Corporate University

106

What Barriers will Prevent you From Implementing ROI?

ROI QuizROI Quiz

107

108

Increasing Response RatesIncreasing Response Rates

• Provide advance communication• Clearly communicate the reason for the questionnaire• Indicate who will see the results• Show how the data will be integrated• Keep the questionnaire simple and brief• Make it easy to respond

• Use the local manager to help distribute the questionnaires and show support

• Let the target audience know that they are part of a carefully selected sample

and . . .

109

• Use one or two follow-up reminders

• Have the introduction letter signed by a top executive

• Enclose a giveaway item with the questionnaire

• Provide an incentive for quick response

• Send a summary of results to target audience

• Distribute questionnaire to a captive audience

• Consider an alternative distribution channel

• Have a third party gather and analyze data.


and . . .

110

• Communicate the time limit

• Consider paying for the time it takes to complete the questionnaire

• Review the questionnaire at the end of the formal session

• Carefully select the survey sample

• Allow completion of the survey during work hours

• Add emotional appeal


and . . .

111

• Design questionnaire to attract attention, with a professional format

• Let participants know what actions will be taken with the data

• Provide options to respond

• Use a local coordinator to help distribute and collect questionnaires

• Frame questions so participants can respond appropriately and make the questions relevant


112

Wisdom of CrowdsWisdom of Crowds

• In this case, the average estimate is near perfect

• Estimates are used everywhere

• Set up your own experiment

• Estimates should be adjusted

• Estimates are okay – defend them; don’t prefer them

113

Action Plan

114

Barriers to ROI include:• Resources

Funding Time People

• Support Managers Colleagues Participants

• Skills• Systems• Culture

115

Actions to Make ROI Work• Adapt versus adopt• Planning and discipline

Establish goals and targets

Assign responsibilities

• Improve needs analysis process

• Implement cost-savings approaches

• Communicate progress

• Develop staff skills ROI Network International ROI Networks Workshops CSTD ROI Network ROI Certification Competency groups Read books/case

studies/articles Develop your own case

study Teach others

116

Key Points to Remember

The Basics:

• ROI is the ultimate measure of profitability of our projects, programs, and processes.

• Reporting ROI alone is insufficient.

• The ROI process develops a balanced set of measures representing a chain of impact.

• Not all programs should be evaluated to ROI.

117

Key Points to Remember

The Barriers:• Resources

• Funding• Time• People

• Support• Managers• Colleagues• Participants

• Skills

118

Key Points to RememberSolutions to Barriers:

• Adapt versus adopt

• Planning and discipline

• Improve needs analysis process

• Implement cost savings approaches

• Communicate progress

• Develop staff skills

119

Key Points to RememberThe Benefits:

• Show the contribution of programs• Earn respect of senior management• Gain the confidence of clients• Improve support for training and performance

improvement• Enhance training and performance improvement

processes• Identify inefficient programs that need to be

redesigned• Identify successful programs

120

What if you do nothing?What if you do nothing?

121

Sample of Published ROI StudiesPerformance Management

Process Improvement

Skill-Based Pay

Sexual Harassment Prevention

Safety Incentive Plan

Diversity

Retention Improvement

Absenteeism Control/Reduction Program

Stress Management Program

Executive Leadership Development

E-Learning

Internal Graduate Degree Program

Executive Coaching

Competency Development

First Level Leadership Development

Southeast Corridor Bank

• Program Title: Managing Retention

• Target Group: Bank Tellers

• Solution: Skills Based Pay System to Reduce Employee Turnover

122

Southeast Corridor Bank• Level 1: Reaction

• Composite Rating: 4.2 out of 5

• Level 2: Learning• Positive self assessment on program

understanding• Positive self assessment on each course with

few exceptions (only two failed to be promoted because of performance in training)

123

• Level 3: Application• 95% participation rate• 86 requests for training compared to 46 the year

before• 138 review situations• 257 promotions compared to 139 the year before

• Level 4: Impact• Turnover reduced from 71% to 35%• Staffing level reduced by 4%

124


• Level 5: ROI • BCR: 3.58• ROI: 258%

• Intangible Benefits• Customer Satisfaction• Job Satisfaction• Product Sales• Cross Selling

125


• Technique to Isolate Effects of Program: Estimates from branch managers and branch staff, adjusted for error

• Technique to Convert Data to Monetary Value: External studies at similar institutions and standard values (for staffing)

• Fully-loaded Program Costs: $857,196 First year; $433,200 Second year

126


127

Case Study Application

128

Retail Merchandise CompanyRetail Merchandise CompanyRetail Merchandise CompanyRetail Merchandise Company

129

Program ProfileProgram Profile

• Title: Interactive Selling Skills• Target Group: Sales Associates in

Electronics• Vendor Produced and Delivered• 3 Days - (2 Days Plus 1 Day)• Significant Use of Skill Practices• 3 Groups Trained (48 Participants from 3

Stores)

130

ROI Analysis ProfileROI Analysis Profile

Converting Data to Monetary ValuesConverting Data to Monetary Values

Isolating the Effects of TrainingIsolating the Effects of Training• Control Group Arrangement• Participant’s Estimate (For Back-up)

• Profit Contribution of Increased Output

• (4) Performance Monitoring 3 months• (3) Questionnaire 3 months• (3) Program Follow-up Session 3 weeks (last session)

Post Program Data CollectionPost Program Data Collection

131

Level 1 - Selected DataLevel 1 - Selected Data

Success with ObjectivesSuccess with Objectives 4.34.3

Relevance of MaterialRelevance of Material 4.44.4

Usefulness of ProgramUsefulness of Program 4.54.5

Exercises/Skill PracticesExercises/Skill Practices 3.93.9

Overall Instructor RatingOverall Instructor Rating 4.14.1

132


All Participants Demonstrated

That They Could Use The Skills Successfully

133


Strongly Agree

Agree Neither Agree Nor Disagree

Disagree Strongly Disagree

78% 22% 0% 0% 0%

With Each Customer

Every Third

Customer

Several Times

Each Day

At Least Once Daily

At Least Once

Weekly

52% 26% 18% 4% 0%

(2 Questions out of 20)

I utilize the skillstaught in the program

Frequency of use ofskills

134

Level 4 Data:Level 4 Data:Average Weekly SalesAverage Weekly Sales

Post Training Data

Weeks After Training Trained Groups Control Groups

1 $ 9,723 $ 9,6982 9,978 9,7203 10,424 9,812

13 $13,690 $11,572 14 11,491 9,683 15 11,044 10,092

Average for Weeks $12,075 $10,449 13, 14, 15

135

Annualized Program BenefitsAnnualized Program Benefits

Average Weekly Sales per Employee Average Weekly Sales per Employee Trained GroupsTrained Groups

$12,075$12,075

Average Weekly Sales per Employee Average Weekly Sales per Employee

Untrained GroupsUntrained Groups

10,44910,449

IncreaseIncrease

1,6261,626

Profit Contribution (2% of Store Sales)Profit Contribution (2% of Store Sales)

32.5032.50

Total Weekly Improvement (x 46)Total Weekly Improvement (x 46)

1,4951,495

Total Annual Benefits (x 48 Weeks)

$71,760

46 participants were still in job after 3 months ..

136

Cost SummaryCost Summary

48 participants in 3 courses

Facilitation Fees: 3 courses @ $3750 Facilitation Fees: 3 courses @ $3750 $11,250 $11,250

Program Materials: 48 @ $35/participant Program Materials: 48 @ $35/participant 1,680 1,680

Meals/Refreshments: Meals/Refreshments: 4,032 4,032

3 days @ $28/participant 3 days @ $28/participant

Facilities: 9 days @ $120Facilities: 9 days @ $120 1,080 1,080

Participant Salaries Plus Benefits (35% factor)Participant Salaries Plus Benefits (35% factor) 12,442 12,442

Coordination/EvaluationCoordination/Evaluation 2,500 2,500

Total Costs $ 32,984

137

Level 5 DataLevel 5 Data

BCR =BCR = ==

ROI (%) =ROI (%) = X 100 =X 100 =

138

ROI Example:ROI Example:Retail Merchandise CompanyRetail Merchandise Company

Collecting Post Program

Data

Isolating the Effects of

the Program

Converting Data to

Monetary Value

Calculating the Return

on Investment

Calculating the Return

on Investment

Tabulating Program

Costs

Identifying Intangible Benefits

$32,984

118%• Follow-up Session

• Questionnaire

• Performance Monitoring

• Control Groups

• Participants’ Estimates

• Standard Values

$71,760

139

The ROI Process Takes A Balanced The ROI Process Takes A Balanced View by Measuring And Reporting:View by Measuring And Reporting:

• Reaction to program• Learning and attitudes• Application on the job• Impact in work unit• Impact on the customer• The financial results• Intangible benefits• Nature and source of problems and opportunities

Documents

1 Show Me The Money Measuring ROI for People, Projects, and Programs Jack J. Phillips, Ph.D