Embed Size (px)
Citation preview
During today’s webcast we will:
Review the 12 basic problems with evaluation.
Learn key features of the 3 steps to building a successful evaluation practice.
Learn how to overcome resistance to evaluation.
Tools for the job:
Pen or pencil
Paper or notebook
Poll Question
Which of the following represents your biggest problem with evaluation?
A. Too many theories and modelsB. Models are too complexC. Lack of understanding of evaluationD. Evaluation is considered a post-program activityE. Lack of support from key stakeholdersF. Improper use of evaluation dataG. Other
There are too many
theories and models. Evaluation is too hard.
I don’t know how to do it!
Evaluation is too subjective
– no statistical precisionEvaluation is post-program only—I need to justify the program before we launch it.
What’s the ROI of
evaluation??Managers aren’t
rewarded to use the data. If the results are negative, it will
impact our bonus.
How do we ensure we compare apples to apples?
I wonder how long we will have to do this.
Three steps build a successful evaluation practice.
Setting the Stage
Developing the Practice
Implementing the Practice
Source: Phillips, J. J., Phillips, P. P., Hodges T. K. (2004) Make Evaluation Work. Alexandria: ASTD
Today we are talking about these components of each step.
Setting the StageDeveloping the Practice
Implementing the Practice
Measurement Culture
Evaluation Readiness
Stakeholder Needs
Evaluation Purpose
Evaluation Model
Expertise
Budgets
Setting the StageSetting the Stage
Does your organization implement the Balanced Scorecard, Six Sigma or any other quality management process?
Are there any type of tracking processes in place such as activity-based costing, cycle time reduction, customer satisfaction measurements?
Do you know the strategic business goals of your organization?
Do you know how the learning function links to these goals?
Setting the StageSetting the Stage Is your organization large with a variety of
programs? Is your organization undergoing significant
change? Is there pressure from senior management to
measure results? Is the image of your training function less than
desirable? Is there an increased focus on linking training to
strategy?
For complete assessment go to Tools & Templates at www.roiinstitute.net.For complete assessment go to Tools & Templates at www.roiinstitute.net.
Action Step 1Action Step 1Stakeholders Stakeholder
Data NeedsCurrent
MeasuresNeeded
Measures
Setting the StageSetting the Stage
Stakeholders Stakeholder Data NeedsTop Executives Impact and ROI data
Scope and volume (indicators)
Client, Sponsors Project approval dataProject results Impact and ROI data
Middle Managers Scope and volume (indicators)Time commitmentsImpact and ROI data
Immediate Managers, Team Leaders
Time requirementsRoles and dutiesApplication and impact data
Setting the StageSetting the Stage
Stakeholders Stakeholder Data Needs
Participants Data and time requirementsFeedback on learningSummary of application and
impact data
Learning and Development Leadership, Program Managers, Coordinators
Summary of reaction, learning, application, impact, ROI data
Designers/Developers Reaction, learning, application, impact, ROI data
Facilitators Reaction, learning, application, impact, ROI data
Evaluator(s) All of the above
Poll Question
Are you providing the measures needed by your stakeholders?
A. Yes
B. No
C. Maybe
Developing the PracticeDeveloping the Practice
What purpose will an evaluation
system serve you, your team,
and your organization?
Developing the PracticeDeveloping the Practice
A successful evaluation system can help you:
• Determine success in achieving program objectives.
• Identify strengths and weaknesses in the learning and development process.
• Set priorities for learning and development resources.
• Test the clarity and validity of tests, cases and exercises.
• Identify the participants who were most (or least) successful with the program.
• Reinforce major points made during the program.
• Decide who should participate in future programs.
• Compare the benefits to the costs of a learning and development program.
• Enhance the accountability of learning and development.
• Assist in marketing future programs.
• Determine if a program was an appropriate solution.
• Establish a database to assist management with decision making.
Write your evaluation purpose statement.
Determine how you know you are successful?
Action Step 2Action Step 2
Purpose drives the initiative to achieve outcomes.
Implementation of our evaluation strategy gives us the information we need to help develop our people, improve our processes, and contribute to the bottom line – literally!
We ensure that taxpayer dollars are allocated to the programs that directly support the agency mission.
Unnecessary programs have been eliminated
Training budget has grown Line supervisors are more supportive Staff development funds have increased Evaluation funds have increased Successful programs are expanding
throughout the enterprise
How will you know you have achieved your purpose?
Developing the PracticeDeveloping the Practice
Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research.
HRD Theory
Psychological TheorySociologySystems TheoryEconomics Theory
HRD Theory
Psychological TheorySociologySystems TheoryEconomics Theory
ADDIE Model
AssessDesignDevelopImplementEvaluate
ADDIE Model
AssessDesignDevelopImplementEvaluate
Evaluation Models
Cost-Benefit AnalysisKirkpatrick’s Four-Level FrameworkPhillips’ Five-Level FrameworkKaufman’s Five Levels of EvaluationUtility AnalysisCIROCIPPMarshall & Schriver’s Model for Evaluating Knowledge and SkillsBusiness Impact ISD ModelSuccess Case EvaluationBrown & Reed’s Integral FrameworkBalanced Scorecard
Evaluation Models
Cost-Benefit AnalysisKirkpatrick’s Four-Level FrameworkPhillips’ Five-Level FrameworkKaufman’s Five Levels of EvaluationUtility AnalysisCIROCIPPMarshall & Schriver’s Model for Evaluating Knowledge and SkillsBusiness Impact ISD ModelSuccess Case EvaluationBrown & Reed’s Integral FrameworkBalanced Scorecard
Phillips, P. P. (2003). Training Evaluation in the Public Sector. Doctoral Dissertation. The University of Southern Mississippi. International Development.
Developing the PracticeDeveloping the Practice
Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research.
Consider the following. Is the process simple? Is the process economical? Is the process credible? Is the process theoretically sound? Is the process based on generally accepted
principles? Does the process account for other factors that
influence outcome measures? Is the process flexible? Is the process applicable with all types of data? Does the process include a mechanism to
develop program costs? Are calculations based on accepted formulas? Does the process have a successful track
record?
Review your current or potential evaluation model; how does it stack up to the following criteria? Simple Economical Credible Theoretically sound Based on generally accepted principles Accounts for other factors Flexible Applicable to all types of data Accounts for all costs Accepted calculations Successful track record
Action Step 3Action Step 3
Implementing the PracticeImplementing the PracticeWhat do you need to know?
Statistical theory and methods
Research design Analysis methods Interpretation and reporting
of data Theories and types of
evaluation
What do you do with what you know?
Identify customer expectations Select or design appropriate
strategies, research design and measures
Communicate and gain support for the evaluation plan
Manage data collection Analyze and interpret data Report conclusions and make
recommendations
Source: Bernthal, P. R., K. Colteryahn, P. Davis, J. Naughton, W. J. Rothwell, and R. Wellins. (2004) ASTD 2004 Competency Study: Mapping the Future, Alexandria: ASTD.
Implementing the PracticeImplementing the Practice
Need Help?•Acquire formal education
•Participate in professional organizations
•Attend conferences
•Attend workshops
•Get Certified in ROI
•Learn on the Job
•Tap into a Network
Implementing the PracticeImplementing the PracticeWhat will it cost?
Salaries & employee benefits for L&D staff (No. of staff x avg. salary x employee benefits factor x no. of hours on evaluation project)
Meals, Travel, and Incidental Expenses Participant Costs (for time involved in evaluation Fees and Licenses Office Supplies and Expense of Printing Materials Outside Services Technology General Overhead Allocation Other Miscellaneous Expenses
3% - 5% of total
training budget
3% - 5% of total
training budget
Poll Question
What if I can’t afford to spend 3-5% of my existing budget on evaluation? My question to you is –
Can you afford NOT to?
A. YesB. NoC. Maybe
Implementing the PracticeImplementing the PracticeAnother way to put it…
Total cost of effective evaluation system 3%-5% of budget Cost of processing one reaction questionnaire $2.00 Application/behavior change study (internal) $2,000 - $3,000 Application/behavior change study (external) $15,000 - $20,000 ROI Impact Study (internal) $5,000 - $10,000 ROI Impact Study (external) $15,000 - $50,000 Fee to attend two-day workshop $850* Fee to attend one-day workshop $450* Fee for ROI Certification $2,995** Cost of Handbook of Training Evaluation and Measurement Methods $62.50 Cost of ROI foldout model Free**
Value of an effective, efficient evaluation system PRICELE$$
*ASTD Member Price **Available at [email protected]
Setting the StageDeveloping the Practice
Implementing the Practice
Recognize your culture and readiness for
measurement as well as your stakeholder data
needs
So that you an define your evaluation purpose
and select the appropriate model.
Then, develop the expertise needed to do
the job with the resources you have
available.
What are the steps to building a successful evaluation practice?
1. Set the StageRecognize your organization’s culture and readiness for measurement along with your stakeholders’ data needs, so that you can
2. Develop the PracticeBy first defining your evaluation purpose and selecting the most appropriate model. Then,
3. Implement the PracticeBy developing the necessary expertise to do the job within your existing resources.
What if “they” resist? Identify a champion Assign roles and responsibilities Establish evaluation targets Develop a project plan for implementation Revise/develop policies and procedures Involve the entire learning staff Teach the staff Select programs for comprehensive evaluation Report progress on studies Prepare the management team
Action Plan
Name: You and Your Team Start Date: February 2, 2005
Project: Making Evaluation Work – An Implementation Plan
Action Steps Completion Date
1. Identify stakeholder data needs. ________________
• Who are our stakeholders?
• What are their data needs?
• What data do we currently measure?
• What is needed?
2. Write evaluation purpose statement. ________________
• How will we know we are successful?
3. Compare evaluation processes to criteria. ________________
Make evaluation
work….
for you!
