Patricia Pulliam Phillips, Ph.D.

and Jack J. Phillips, Ph.D.

The Value of LearningHow Organizations Capture Valueand ROI and Translate Them intoSupport, Improvement, and Funds

Q

John Wiley & Sons, Inc.

Phillips.ffirs 6/3/07 1:07 PM Page vii

File AttachmentC1.jpg

Phillips.ffirs 6/3/07 1:07 PM Page vi

About This Book

Why is this topic important?While other books may be narrowly focused or represent unprovenapproaches, The Value of Learning presents a methodical approach that canbe replicated throughout an organization, enabling comparisons of resultsfrom one program to another. The process described in this book is the mostdocumented method in the world, and its implementation has been phe-nomenal, with over four thousand individuals participating in five-day cer-tification programs designed for its implementation. While other books maydelve into accountability in certain types of programs or data, this bookshows a method that works across all types of programs, ranging from lead-ership development to basic skills training for new employees. With thisapproach, every program is evaluated at some level.

What can you achieve with this book?A Guide to Learning Value

This book is a basic guide for anyone involved in implementing learning anddevelopment programs. Strategies that will assist in collecting data duringand after program implementation are presented. This book addresses aresults-based approach to program implementation, focusing on a variety ofmeasures, categorized into seven data types:

1. Inputs and Indicators2. Reaction and Planned Action3. Learning and Confidence4. Application and Implementation5. Impact and Consequences6. Return on Investment7. Intangibles

The Value of Learning helps learning professionals identify, collect, ana-lyze, and report all seven types of data in a consistent manner that ensurescredible results.

Phillips.ffirs 6/3/07 1:07 PM Page i

Credibility Is Key

The Value of Learning focuses on building a credible processone that gen-erates a value that is believable, realistic, and accurate, particularly from theviewpoint of sponsors and stakeholders. More specifically, the methodologypresented in this book approaches credibility head-on through the use of:

Balanced categories of data A logical systematic process Guiding principles, a conservative set of standards A proven methodology based on thousands of applications and uses An emphasis on implementing the methodology within an

organization so that the process is sustained A process accepted by sponsors, clients, and others who fund

projects and programs

The book explores the challenges of collecting hard-to-measure dataand placing monetary values on the hard-to-value. Building on a tremen-dous amount of experience, application, practice, and research, the bookdraws on the work of many individuals and organizations, particularly thosewho have been reaching the ultimate levels of accountability, the ROI. Devel-oped in an easy-to-read format and fortified with examples and tips, thiswill be an indispensable guide for audiences who seek to understand moreabout bottom-line accountability.

How is this book organized?The book follows the value chain of learning and development as the mea-sures move through the learning and development cycle. The first two chap-ters set the stage for the entire process. Chapter One, Building aComprehensive Evaluation Process, illustrates the challenges and opportuni-ties for evaluating learning, outlining some of the progress made and chal-lenges we face to bring more accountability to learning and development.This chapter also introduces the entire process explained in the book, detail-ing the types and levels of data, the systematic step-by-step method, andthe standards and principles used throughout the book.

Chapter Two, Defining Needs and Objectives: Ensuring Business Align-ment, describes the beginning point of any learning and development

Phillips.ffirs 6/3/07 1:07 PM Page ii

process. The focus of this chapter is identifying the need. Here, practical tech-niques and tips are provided to ensure that the program is aligned with theorganization, particularly the business alignment link. It is at this stage thatmany programs go astray. Also this chapter shows how to develop objectivesat different levels, moving beyond the traditional learning objectives todevelop application, impact, and even ROI objectives.

Chapter Three, Measuring Inputs and Indicators, discusses the first ofthe data categories with a focus on inputs and indicators. Tracking the inputsto the process is a basic necessity, measuring data such as number of peo-ple, number of hours, and number of programs, as well as costs and effi-ciencies. This level of data is important, but does not show the value of theprograms.

Chapter Four, Measuring Reaction and Planned Action, addresses thefirst level of results based on perceptions. This chapter shows how measure-ment at this fundamental level has been changed and altered recently toimprove its value.

Next, Chapter Five, Measuring Learning and Confidence, focuses onmeasuring learning, the heart of any evaluation process for learning anddevelopment. It shows how learning measures are being addressed in amethodical and efficient way to ensure that participants leave programs hav-ing acquired the skills and knowledge necessary to take action in their orga-nizations.

Chapter Six, Measuring Application and Implementation, measures thefirst level beyond the actual program. Essentially, data collected at this levelanswer a question: What are participants doing differently as a result of theprogram? This chapter presents ways in which this follow-up data can becaptured and particular measures that must be monitored.

Chapter Seven, Measuring and Isolating the Impact of Programs, movesto impacts and consequences, which are key data that executives want tosee. This chapter connects the learning program to the business, showinghow the data are collected and how the impact of the program is isolatedshowing how much of the impact data are actually from other factors.

Chapter Eight, Identifying Benefits and Costs, and Calculating ROI,shows how the ROI is developed and how the business impact measures areconverted to monetary values. It also details how the costs of the programare tabulated so that a comparison can be made between the benefits andthe costs of the program. For some, this is the ultimate level of evaluation.

Phillips.ffirs 6/3/07 1:07 PM Page iii

Chapter Nine, Measuring the Hard to Measure and the Hard to Value:Intangible Benefits, addresses the important issue of intangibles. The intan-gible benefits are those measures that are purposely not converted to mon-etary values. They must be monitored properly, analyzed appropriately, andreported credibly.

Chapter Ten, Reporting Results, focuses on how to report data to anumber of target audiences, illustrating some of the most efficient and effec-tive ways to communicate data.

Finally, Chapter Eleven, Implementing and Sustaining a ComprehensiveEvaluation System, provides detail on how to support and sustain this eval-uation process. It addresses particular issues of how to overcome the resis-tance of the staff and get the management team more involved in theevaluation process.

Phillips.ffirs 6/3/07 1:07 PM Page iv

About PfeifferPfeiffer serves the professional development and hands-on resource needsof training and human resource practitioners and gives them products to dotheir jobs better. We deliver proven ideas and solutions from experts in HRdevelopment and HR management, and we offer effective and customiz-able tools to improve workplace performance. From novice to seasonedprofessional, Pfeiffer is the source you can trust to make yourself and yourorganization more successful.

Essential Knowledge Pfeiffer produces insightful, practical, andcomprehensive materials on topics that matter the most to training

and HR professionals. Our Essential Knowledge resources translate theexpertise of seasoned professionals into practical, how-to guidance on crit-ical workplace issues and problems. These resources are supported by casestudies, worksheets, and job aids and are frequently supplemented with CD-ROMs, Web sites, and other means of making the content easier to read,understand, and use.

Essential Tools Pfeiffers Essential Tools resources save time andexpense by offering proven, ready-to-use materialsincluding exer-

cises, activities, games, instruments, and assessmentsfor use during atraining or team-learning event. These resources are frequently offered inlooseleaf or CD-ROM format to facilitate copying and customization of thematerial.

Pfeiffer also recognizes the remarkable power of new technologies inexpanding the reach and effectiveness of training. While e-hype has oftencreated whizbang solutions in search of a problem, we are dedicated tobringing convenience and enhancements to proven training solutions. Allour e-tools comply with rigorous functionality standards. The most appro-priate technology wrapped around essential content yields the perfect so-lution for todays on-the-go trainers and human resource professionals.

Essential resources for training and HR professionalsw w w. p f e i f f e r . c o m

Phillips.ffirs 6/3/07 1:07 PM Page v

This book is dedicated to our loving parents,

who taught us more about value than we could

ever appreciate:

Flo S. Pulliam

Benjamin F. Pulliam

Agnes B. Phillips

George W. Phillips

Phillips.ffirs 6/3/07 1:07 PM Page vi

Patricia Pulliam Phillips, Ph.D.

and Jack J. Phillips, Ph.D.

The Value of LearningHow Organizations Capture Valueand ROI and Translate Them intoSupport, Improvement, and Funds

Q

John Wiley & Sons, Inc.

Phillips.ffirs 6/3/07 1:07 PM Page vii

Copyright 2007 by John Wiley & Sons, Inc. All Rights Reserved.

Published by PfeifferAn Imprint of Wiley989 Market Street, San Francisco, CA 94103-1741www.pfeiffer.com

Wiley Bicentennial logo: Richard J. Pacifico

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, orotherwise, except as permitted under Section 107 or 108 of the 1976 United States Copy-right Act, without either the prior written permission of the Publisher, or authorizationthrough payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc.,222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to thePermissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030,201-748-6011, fax 201-748-6008, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used theirbest efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim anyimplied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice andstrategies contained herein may not be suitable for your situation. You should consult with aprofessional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental,consequential, or other damages.

Readers should be aware that Internet websites offered as citations and/or sources for furtherinformation may have changed or disappeared between the time this was written and when itis read.

For additional copies/bulk purchases of this book in the U.S. please contact 800-274-4434.

Pfeiffer books and products are available through most bookstores. To contact Pfeiffer directlycall our Customer Care Department within the U.S. at 800-274-4434, outside the U.S. at 317-572-3985, fax 317-572-4002, or visit www.pfeiffer.com.

Pfeiffer also publishes its books in a variety of electronic formats. Some content that appearsin print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data

Phillips, Patricia Pulliam.The value of learning : how organizations capture value and ROI and translate them into

support, improvement, and funds / Patricia Pulliam Phillips and Jack J. Phillips.p. cm.

Includes bibliographical references and index.ISBN 978-0-7879-8532-5 (cloth)

1. EmployeesTraining ofEvaluation. 2. EmployeesTraining ofCost effectiveness.3. Rate of return. I. Phillips, Jack J., 1945 II. Title.

HF5549.5.T7P439 2007658.3'124dc22 2007011227

Acquiring Editor: Matthew Davis Production Editor: Michael KayDirector of Development: Kathleen Dolan Davies Editor: Rebecca TaffEditorial Assistant: Julie Rodriguez Manufacturing Supervisor: Becky Morgan

Printed in the United States of AmericaPrinting 10 9 8 7 6 5 4 3 2 1

Phillips.ffirs 6/3/07 1:07 PM Page viii

www.pfeiffer.comwww.pfeiffer.com

Contents

List of Exhibits, Figures, and Tables xxi

Preface xxvii

Acknowledgments xxxi

Chapter One: Building a Comprehensive Evaluation Process 1

Challenges 2

Global Evaluation Trends Measurement and Evaluation Challenges Benefits of Measurement and Evaluation The Myths of Measurement and Evaluation

Key Steps and Issues 13

Stakeholders Levels and Steps Chain of Impact ROI Process Model Objectives Evaluation Planning Data Collection Analysis Isolation of the Effects of Learning and Development Conversion of Data to Monetary Values The Cost of Programs The Return on Investment Calculation Intangible Benefits DataReporting Operating Standards Implementation Issues

Final Thoughts 31

Chapter Two: Defining Needs and Objectives:Ensuring Business Alignment 32

The Challenge 32

Business Alignment Issues Begin with the End in Mind Required Discipline The Needs Analysis Dilemma

ix

Q

Phillips.ftoc 6/3/07 1:08 PM Page ix

Payoff Needs 36

Key Questions Obvious vs. Not So Obvious The Reasons for New Programs Determining Costs of the Problem The Value of Opportunity To Forecast or Not to Forecast

Business Needs 44

Determining the Opportunity Defining the Business MeasureHard Data Defining the Business NeedSoft Data Using Tangible vs. IntangibleA Better Approach Finding Sources of Impact Data Identifying All the Measures Exploring What If. . . ?

Job Performance Needs 53

Analysis Techniques Taking a Sensible Approach

Learning Needs 58

Subject-Matter Experts Job and Task Analysis Observations Demonstrations Tests Management Assessment

Preference Needs 61

Key Issues Impact Studies

Levels of Objectives for Programs 65

Reaction and Planned Action Learning Objectives Application and Implementation Objectives Business Impact Objectives ROI Objectives The Importance of Specific Objectives

Final Thoughts 73

Chapter Three: Measuring Inputs and Indicators 74

Measuring Input and Indicators 75

Defines the Input Reflects Commitment Facilitates Benchmarking Explains Coverage Highlights Efficiencies Provides Cost Data

Tracking Participants 78

Tracking Hours 80

Tracking Coverage by Jobs and Functional Areas 81

Tracking Topics and Programs 82

x CONTENTS

Phillips.ftoc 6/3/07 1:08 PM Page x

Tracking Requests 84

Tracking Delivery 85

Tracking Costs 86

Pressure to Disclose All Costs The Danger of Costs Without Benefits Sources of Costs Learning Program Steps and Costs Prorated Versus Direct Costs Employee Benefits Factor Major Cost Categories Cost Reporting

Tracking Efficiencies 94

Tracking Outsourcing 95

Tracking for the Scorecard 96

Defining Key Issues 97

Input Is Not Results Reports to Executives Should Be Minimized The Data Represent Operational Concerns This Data Must Be Automated

Final Thoughts 98

Chapter Four: Measuring Reaction and Planned Action 100

Why Measure Reaction and Planned Action? 101

Customer Service Early Feedback Is Essential Making Adjustments and Changes Predictive Capability For Some, This Is the Most Important Data Comparing Data with Other Programs Creating a Macro Scorecard

Sources of Data 106

Participants Participants Managers Internal Customers Facilitators Sponsors/Senior Managers

Areas of Feedback 107

Content vs. Non-Content The Deceptive Feedback Cycle Key Areas for Feedback

Timing of Data Collection 114

Early, Detailed Feedback Pre-Assessments Collecting at Periodic Intervals For Long Programs with Multiple Parts

Contents xi

Phillips.ftoc 6/3/07 1:08 PM Page xi

Data Collection with Questionnaires and Surveys 115Questionnaire/Survey Design Intensities Questionnaire/Survey Response Rates Sample Surveys

Data Collection with Interviews and Focus Groups 123

Improving Reaction Evaluation 123Keep Responses Anonymous Have a Neutral Person Collect the Forms Provide a Copy in Advance Explain the Purpose of the Feedback and How It Will Be Used Explore an Ongoing Evaluation Consider Quantifying Course Ratings Collect Information Related to Improvement Allow Ample Time for Completing the Form Delayed Evaluation Ask for Honest Feedback

Using Data 127Building the Macro-Level Scorecard

Shortcut Ways to Measure Reaction and Planned Action 129

Final Thoughts 130

Chapter Five: Measuring Learning and Confidence 132

Why Measure Learning and Confidence? 132The Importance of Intellectual Capital The Learning Organization The Learning Transfer Problem The Compliance Issue The Use and Development ofCompetencies The Role of Learning in Programs The Chain of Impact Certification Consequences of an Unprepared Workforce

The Challenges and Benefits of Measuring Learning 137The Challenges The Benefits

Measurement Issues 140Objectives Typical Measures Timing Cognitive Levels of Blooms Taxonomy

Data Collecton Methods 144Questionnaires/Surveys Objective Tests Criterion-Referenced Tests Performance Tests Technology and Task Simulations Case Studies Role Playing/Skill Practice Assessment Center Method Exercises/Activities Informal Assessments

xii CONTENTS

Phillips.ftoc 6/3/07 1:08 PM Page xii

Administrative Issues 157

Reliability and Validity Consistency Monitoring Pilot Testing Readability Scoring Reporting Confronting Test Failures

Using Learning Data 161

Final Thoughts 162

Chapter Six: Measuring Application and Implementation 163

Why Measure Application and Implementation? 163

The Value of Information A Key Transition Time The Key Focus of Many Programs The Chain ofImpact Barriers and Enablers Reward Those Who Are Most Effective

Challenges of Measuring Application and Implementation 166

Linking Application with Learning Designing Data Collection into Programs Applying Serious Effort to Level 3 Evaluation Including Level 3 in the Needs Assessment Developing ROI with Application Data

Key Issues 168

Methods Objectives Topics to Explore Sources Timing Responsibilities

The Use of Questionnaires 172

Progress with Objectives Relevance/Importance ofthe Program Knowledge/Skill Use Changes withWork/Action Items Improvements/Accomplishments Define the Measure Provide the Change Monetary Value Total Impact List of Other Factors Improvements Linked with the Program Perceived Value Links with Output Measures Success of the Program Team Barriers and Enablers ManagementSupport Appropriateness of Program and Suggestions forImprovement Checklist Improving Response Rates

Data Collection with Interviews 189

Types of Interviews Interview Guidelines

Data Collection with Focus Groups 191

Applications for Focus Group Evaluation Guidelines

Contents xiii

Phillips.ftoc 6/3/07 1:08 PM Page xiii