Upload
dmdk12
View
2.734
Download
0
Tags:
Embed Size (px)
DESCRIPTION
ROI of Training - Isolating the Results
Citation preview
ROI OF TRAININGROI OF TRAINING
Session 6 – Isolating the Session 6 – Isolating the Results/Effects Of the Training Results/Effects Of the Training InterventionIntervention
Pre-Amble
Isolating the results/effect of the training effort is the most difficult and also most important element of building the case for value (ROI);
Because of this, value at the higher levels is seldom determined;
However, the credibility not only of your study to show value, but also the training function is based on this element.
Pre-Amble
The complexity of isolating the effect, is a function of the variables that are in play at any moment in time in an organization
Pre-Amble
The Challenge
Pre-Amble
An acceptable degree of accuracy
External Factors
ManagementAttention
Incentives
Systems/ProceduresChanges
LearningProgram
TOTALIMPROVEMENT
AFTERPROGRAM
EFFECT OFLEARNING ON IMPROVEMENT
How then do you isolate the
effects?
Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors
Isolating the Effects of the Intervention
Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors
Isolating the Effects of the Intervention
Isolating the Effects of the Intervention
External Factors
ManagementAttention
Incentives
Systems/ProceduresChanges
LearningProgram
ELIMINATE
Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors
Isolating the Effects of the Intervention
Isolating the Effects of the Intervention
External Factors
ManagementAttention
Incentives
Systems/ProceduresChanges
LearningProgram
CALCULATEAND ATTRIBUTE THE
REST
Control Groups
Most accurate isolation approach; Involves the use of an experimental group
that is impacted by the intervention and a control group that is not;
Composition of both groups should be as identical as possible and, if feasible, the selection of participants for each group should be on a random basis.
Control Group Designs
Control GroupControl Group
ExperimentalGroup
M1M1
M1 Program
M2M2
M2
Question is What Do You
Control?
Case Study
Federal Express ROI analysis using control groups as a good example of the state of the art in measuring ROI. The study focused on 20 employees who went through an intense, redesigned two-week training program soon after being hired to drive company vans. Their performance was compared with a control group of 20 other new hires whose managers were told to do no more or less on-the-job training than they normally would. Performance was tracked for the two groups for 90 days in categories such as accidents, injuries, time-card errors, and domestic air-bill errors. The ten performance categories were assigned monetary values by experts from engineering, finance, and other groups. The intervention demonstrated that the performance of the highly trained employees was superior to that of the group that did not receive the upgraded training and also resulted in a 24% return on investment.
Also look at CLC Best PracticeDocument
Federal Express demonstrates: Practicality Non disruptive; Relatively accurate; Line involvement (and commitment); ROI is possible if you know what you are
aiming for (measures)
Also look at CLC Best PracticeDocument
Practical Exercise
Accra Wholesale Brickworks, find that there production targets are not being met. The Company is midsize but established in the City. It employs a production team consisting of three foreman, that supervise approximately 8 workers each and these teams take responsibility for procurement of the raw materials, mixing, filling the moulds, drying and baking. The production function is managed by a section manager.
Practical Exercise
Practical Exercise
Using the control group method: How would you selects your experimental
group vis your control group What will you control How will you apply the Trend line
Analysis Technique?
Trend-line Analysis
Basic format less accurate than control group Draw a trend-line using previous performance data
as a base When intervention is conducted, actual
performance is plotted and compared to the trend line
Improvement in actual performance over what the trend line predicted can then be reasonably attributed to the intervention
The primary advantage of this approach is that it is simple, inexpensive, and takes very little effort;
However not accurate as it does not isolate into other factors;
However a good first order analysis.
AveragePost Program 35
AveragePre Program 55
Time
Com
pla
ints
SexualHarassment
Prevention Program
Formal Internal Complaints of Sexual Harassment
O N D J F M A M J J A S O N D J F M A M J J A S O
Trend-line Analysis
Trend line Use Case Study(Micro Electronics Incorporated)
Practical Exercise
Estimate of Impact by Key Role Players
Easy to implement; Based on the assumption that participants
are capable of determining or estimating how much of a performance improvement/impact is related to the intervention;
Because their actions have produced the improvement, participants may have very accurate input on the issue;
Although an estimate, this value will usually have considerable credibility with management because participants are at the center of the change or improvement;
High face validity.
Estimate of Impact by Key Role Players
Easy to implement; Based on the assumption
that participants are capable of determining or estimating how much of a performance improvement/impact is related to the intervention;
Because their actions have produced the improvement, participants may have very accurate input on the issue;
Although an estimate, this value will usually have considerable credibility with management because participants are at the center of the change or improvement;
High face validity.
TypicalResearch Questions
The individuals who do not respond to the questionnaire or provide non- usable data on the questionnaire are assumed to report no improvement/impact;
Extreme data and unrealistic claims are omitted from the analysis;
Only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation;
The confidence level, expressed as a percent, is multiplied by the improvement value to reduce the amount of the improvement by the potential error;
If value is expressed it is also factored by the confidence value.
Adjustments to
improve validity
Estimate of Impact by Key Role Players
Estimate of Impact by Key Role Players
% Program Contributed to Improvement
Confidence of Statement as %
Factor Total Improvement (annualized)– Value
Contribution to Improvement
60 % (.6) 40% (.4) .24 1000 240
60 % (.6) 40% (.4) .24 1000 240
50% (.5) 50% (.5) .25 1000 250
Estimate of Impact by Key Role Players
The process can be
expandedto
include:
Supervisors Management Other stakeholders
Expert Input
Select carefully, based on their knowledge of the process and situation;
If capacity does not exist in the organization; Will probably use techniques that most
training practitioners can implement; Organization culture will determine whether
the outcome will be accepted; If linked to skills transfer it could be
worthwhile; Often top management place more confidence
in outside experts than in their own staff.
Obtaining Input From a Target Group
An example is the subordinates of the supervisors trained in people handling skills;
It could be possible for them to estimate how much of an improvement can be attributed to the intervention;
They can also provide input in terms of what other factors might have contributed to the improvement;
The input is usually obtained through surveys or interviews. When the survey results show significant changes in participant behavior after training and no significant change in the general work climate, the improvement in work performance, therefore, must be attributed to the changes in supervisor behavior since other factors remained constant;
This approach has some disadvantages. Data from subordinates is subjective and may be questionable because of the possibility for biased input;
Often subordinates may have difficulty in determining changes in the work climate. However, in some cases, subordinates are aware of the factors that caused changes in their work unit and can provide input about the magnitude or quantity of changes. When combined with other methods to isolate impact, this process has increased credibility.
Obtaining Input From a Target Group
In some situations it could be feasible to calculate the impact of factors (other than the intervention) that influenced the improvement and then conclude that the intervention is credited with the remaining portion;
In terms of this approach, the intervention claims credit for improvement that cannot be attributed to other factors;
The method is appropriate when the other factors are easily identified and the appropriate mechanisms are in place to calculate their impact on the improvement;
The process can be very credible if the method used to isolate the impact of other factors is credible.
Which Strategy?
Control group feasibility; Availability of historical
data; Expertise to conduct study; Capability and willingness
to estimate impact; Can non-training effects be
isolated? What are the constraints in
terms of using the data collection technique and isolation strategy.
Credibility of Data is Influenced By:
Reputation of the source of the data;
Reputation of the source of the study;
Motives of the researchers; Personal bias of audience; Methodology of the study; Assumptions made in the
analysis; Realism of the outcome data; Type of data; Scope of analysis.