Pa#  Phillips,  Ph.D.  pa#@roiins1tute.net  

ROI  FOR  PEOPLE  WHO  (THINK)  THEY  HATE  ROI!  

Objec1ves  

  Calculate ROI, given monetary benefits and program costs

  Name the five levels of evaluation   Name the ten steps in the ROI

Methodology   Explain the need for the 12 guiding

principles

Do  you  agree  with  the  following?  

Top  5  Reasons  People  (Think)  They  Hate  ROI  

1. ROI  formula  is  too  complicated  

2. ROI  process  generates  only  one  type  of  data  3. ROI  cannot  be  applied  to  soU  skills  4. ROI  will  be  requested  for  every  program  

5. ROI  consumes  too  many  resources  

The  ROI  calcula1on  is  simple.  

BCR =

ROI =

Program Benefits Program Costs

Net Program Benefits Program Costs

X 100

Try  it!  

$750,000 $425,000 BCR =

$750,000 - $425,000 $425,000

ROI = X 100

The  ROI  Calcula1on  

$750,000 $425,000 BCR =

$750,000 - $425,000 $425,000

ROI = X 100

= 1.76:1

= 76%

What  Makes  a  Good  ROI?  

1.  Set  the  value  at  the  same  level  as  other  investments  –  15%  

2.  Set  slightly  above  other  investments  –  25%  

3.  Set  at  break  even  –  0%  4.  Set  at  client  

expecta1ons  

While  we  don’t  strive  for  a  nega1ve  ROI,  nega1ve  is  not  

always  bad!  

An Evaluation Framework

Case Applications and Practice

A Process Model

Implementation

To  make  ROI  work,  five  pieces  of  the    puzzle  come  together.  

Operating Standards and

Philosophy

Levels of Evaluation Measurement Focus

0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing

1. Reaction & Perceived Value

Reaction to the project or program, including the perceived value

Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action

2. Learning & Confidence

Learning to use the content and materials, including the confidence to use what was learned

Skills, Knowledge, Capacity, Competencies, Confidence, Contacts

3. Application & Implementation

Use of content and materials in the work environment, including progress with actual items and implementation

Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use

4. Impact and Consequences

The consequences of the use of the content and materials expressed as business impact measures

Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement

Typical Measures

5. ROI Comparison of monetary benefits from program to program costs

Benefit-Cost Ratio (BCR), ROI%, Payback Period

The  levels  serve  three  purposes!  

Evalua1on  

Objec1ves  

Needs  

The  Alignment  Process  

Start  Here  

End  Here  

5                                                            ROI  Objec1ves                                                                                                                          5  

4                                        Impact  Objec1ves                                      4  

3                                    Applica1on  Objec1ves                        3      

2                                                      Learning  Objec1ves                2  

1                        Reac1on  Objec1ves        1  

 Project  

Payoff  Needs  

Business  Needs    

Job  Performance  Needs    

Learning    Needs    

Preference  Needs    

Impact  

Learning  

Reac1on  

Ini1al    Analysis  

Measurement  and  Evalua1on  

Applica1on  

Business  Alignment  and  Forecas1ng   The  ROI  Process  Model  

ROI  Methodology  

Develop Evaluation Plans and

Baseline Data

Develop Objectives Of Solution

Stage 1 Evaluation Planning

Stage 2 Data Collection

Collect Data During Solution Implementation

Level 1

Level 2

Collect Data After Solution

Implementation

Level 3

Level 4

Capture Costs

Of Solution

Isolate the Effects of Solution

Convert Data to Monetary

Value

Stage 4 Communicate

Results Stage 3

Data Analysis

Calculate the Return On

Investment

Level 5

Identify Intangibles Measures

Intangible Benefits

14

Operating Standards 1.  Report the complete story

2.  Conserve resources

3.  Use the most credible sources

4.  Choose the most conservative

alternatives

5.  Isolate the effects of the program

6.  No data no improvement

7.  Adjust estimates for error

8.  Throw out the extreme and unsupported

9.  Use first year benefits for short-term

programs

10. Include fully-loaded costs

11. Report intangible benefits

12. Communicate results to all stakeholders

Reac1on  

Learning  

Applica1on  

Impact  

ROI  

Isolate  the  Effects  of  the  Program  

Intangible  Benefits   aaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Inputs  

Only  a  select  few  programs  are  evaluated  to  ROI.  Consider  the  following:  •  Life  cycle  of  the  program  •  Linkage  of  program  to  opera1onal  goals  and  issues  •  Importance  of  program  to  strategic  objec1ves  

•  Top  administrator  interest  in  the  evalua1on  •  Cost  of  the  program  •  Visibility  of  the  program  •  Size  of  target  audience  •  Investment  of  1me  required  

Southeast  Corridor  Bank   Program  

 Skills  Based  Pay  Systems  

 Business  Impact     Employee  Turnover   Staffing  Level   Customer  and  Job  Sa1sfac1on   Product  Sales   Cross  Selling  

Sprint/Nextel   Program  

 All-­‐Inclusive  Workforce  Program  (AIW)  

 Business  Impact   Akri1on  Rate     Employee  Sa1sfac1on   Communica1on   Coopera1on  &  Teamwork   Diversity  Mix  

Health  Systems   Program  

 Sexual  Harassment  Preven1on  Workshop  

 Business  Impact   Turnover  Reduc1on   Complaint  Reduc1on   Job  Sa1sfac1on   Absenteeism   Stress  Reduc1on   Recrui1ng  

Verizon  Communica1ons   Program  

 Customer  Service  Training  

 Business  Impact  (Target)   Customer  Complaint  Escala1on  

•  Follow-­‐Up  Surveys  •  Follow-­‐Up  Ques1onnaires  •  Observa1on  On  the  Job  •  Interviews  with  Par1cipants  •  Follow-­‐Up  Focus  Groups  •  Program  Assignments  •  Ac1on  Planning  •  Performance  Contrac1ng  •  Project  Follow-­‐Up  Session  •  Performance  Monitoring  

Level 3

Level 4

Collec1ng  Post  Program  Data  

Factors  to  consider  when  selec1ng  

Data  Collec*on  Method   Timing  of  Data  Collec*on  

•  Type  of  data  •  Time  –  Par1cipant  /  Supervisor  

•  Costs  •  Accuracy  –  Validity  /  Reliability  

•  U1lity  •  Culture  /  Philosophy  

•  Availability  of  data  •  Ideal  1me  for  behavior  change  (Level  3)  

•  Ideal  1me  for  business  impact  (Level  4)  

•  Convenience  of  data  collec1on  

•  Constraints  on  data  collec1on  

•  Use  of  a  control  group  arrangement  

•  Trend  line  analysis  of  performance  data  

•  Use  of  forecas1ng  methods  of  performance  data  

•  Par1cipant’s  es1mate  of  program  impact  (percent)  

•  Supervisor’s  es1mate  of  program  impact  (percent)  

•  Manager’s  es1mate  of  program  impact  

•  Use  of  expert/previous  studies  

•  Calculate/es1mate  the  impact  of  other  factors  

•  Customer  input  

Methods  to  Isolate  Program  Effects  

Control  Group  Method  Design  What  is  the  difference  in  improvement?  

Experimental Group M1 Program M2

Control Group M1 M2

Control  Group  Method  Design  What  is  the  difference?  

Control Group

Experimental Group M1

M1

Program

Control  Group  Method  Design  Does  exposure  to  the  measure  make  a  difference?  

Experimental

Control

Program

Experimental (No pre-measure)

M1 M2

M1 M2

Program M1

2%  

1%  

J F M A M J J A S O N D J

ERROR  RATE  

1.85%  Pre  Program  Average  

1.45%  Projected  Average  

MONTHS  

CPI  Program  Conducted  

.7%  Post  Program  Average  

Example  of  Es1ma1on  Monthly increase in credit card accounts: 175 (fact)

Contributing Factors

Consensus Impact (%)

Average Confidence (%)

Sales Training 32% 83%

Incentives 41% 87%

Management Reinforcement

14% 62%

Market Fluctuations 11% 91%

Other _________ 2% 91% 100%

Es1ma1ons  Influence   Fact   %  Contrib.   Est.    Impact   Confidence   Adjusted  

Impact  

Sales  Training  

175     32%   56   83%   46.48  

Fact: 175 New Credit Card Accounts

% Contribution: 32%

Est. Impact: 56

Uncertainty 17%

Margin of Error: +/- 9.52

65.52

56

46.48

Data  are  converted  by:  

•  Converting output to contribution – standard value •  Converting the cost of quality – standard value •  Converting employee’s time – standard value •  Using historical costs •  Using internal and external experts •  Using data from external databases •  Linking with other measures •  Using participants’ estimates •  Using supervisors’ and managers’ estimates •  Using staff estimates

Example:  Cost  of  One  Turnover    from  External  Database  

*  Value  obtained  from  industry-­‐related  study  (external  data)  

• $70,  000/annually  

Salary  of  Middle  Manager  

• 150%  of  annual  Salary  

Value  of  Turnover*  

• $105,000  

Cost  of  Turnover  

Cost  of  A  Sexual  Harassment  Complaint  

$852,000  35  

5  Steps  to  Data  Conversion  

Step  1:  Focus  on  a  unit  of  measure  

Step  2:  Determine  the  value  (V)  of  each  unit  

Step  3:  Calculate  the  change  in  performance  (∆P)  

Step  4:  Determine  the  annual  amount  of  change  (A∆P)  

Step  5:  Calculate  the  total  annual  value  of  the  improvement  (A∆P  x  V)  

Example  Using  Internal  Experts  

Step  1:  One  grievance  

Step  2:  V  =  $6,500  (from  Director  of  nursing  and  HR  Experts)  

Step  3:  ∆P  =  average  of  7    out  of  10  grievances  prevented  per  month  

Step  4:  Annual  ∆P  =  

Step  5:  A∆P  x  V  =  

Example  Using  Internal  Experts  

Step  1:  One  grievance  

Step  2:  V  =  $6,500  (from  Director  of  nursing  and  HR  Experts)  

Step  3:  ∆P  =  average  of  7    out  of  10  grievances  prevented  per  month  

Step  4:  Annual  ∆P  =  7  x  12  =  84  

Step  5:  A∆P  x  V  =  84  x  $6,500  =  $546,000  

Example  Using  Standard  Values  

Step  1:  One  sale  

Step  2:  V  =  30%  profit  margin  

Step  3:  ∆P  =  $20,000  revenue  per  month  

Step  4:  Annual  ∆P  =  

Step  5:  A∆P  x  V  =  

Example  Using  Standard  Values  

Step  1:  One  sale  

Step  2:  V  =  30%  profit  margin  

Step  3:  ∆P  =  $20,000  revenue  per  month  

Step  4:  Annual  ∆P  =  $20,000  x  12  =  $240,000    

Step  5:  A∆P  x  V  =  $240,000  x  .30  =  $72,000  

Why  does  the  ROI  Methodology  work?  

•  Balanced  set  of  measures  •  Step-­‐by-­‐step  process  •  Bridges  evalua1on  disciplines  •  Balances  research  and  reality  •  Flexible  •  Credible  

Next  Steps  

Assess  your  readiness  for  ROI  

Iden1fy  stakeholders  and  their  data  needs  

Determine  the  purpose  of  your  evalua1on  prac1ce  

Iden1fy  programs  suitable  for  ROI  

Develop  capability  in  the  ROI  Methodology  

Pa#  Phillips,  Ph.D.  pa#@roiins1tute.net  

ROI  FOR  PEOPLE  WHO  (THINK)  THEY  HATE  ROI!