Effectiveness of the Intervention Product Service Systems ... · 3.0, Effectiveness of the Intervention. Section 1 – Topic, Audience and Purpose (TAP) and Section 2, Problem will

1

Effectiveness of the Intervention October 14th, 2011

National Product Service Center

Rent-A-Center, Inc.

Table of Contents:

Introduction ........................................................................................................................... 2

1.0 TAP (Topic, Audience & Purpose) ................................................................................... 2

1.2 Topic ........................................................................................................................... 3 1.3 Audience ..................................................................................................................... 3 1.4 Purpose ....................................................................................................................... 3 1.5 Format ......................................................................................................................... 3

2.0 Problem .......................................................................................................................... 3

2.1 Learners ...................................................................................................................... 4 2.2 Instructors .................................................................................................................... 4 2.3 Core problems with existing instructional design .......................................................... 4 2.4 Overview of existing skills, knowledge and limitations.................................................. 4

3.0 Effectiveness of the Intervention ..................................................................................... 5

3.1 Training Content .......................................................................................................... 5 3.2 Training Design ........................................................................................................... 5 3.3. Training Instructor (Facilitator) .................................................................................... 6 3.4 Training Results ........................................................................................................... 6 3.5 Summary ..................................................................................................................... 6

Appendix A. Training Evaluation Questionnaire .................................................................... 8

Appendix B. Training Evaluation Questionnaire Results ..................................................... 10

Appendix C. Kirkpatrick’s Four Levels of Evaluation .......................................................... 11

Figure 1. Instructional System Design Concept Map ...................................................... 12

References ......................................................................................................................... 13

Effectiveness of the Intervention – Product Service Systems Orientation

Tony F. Fuller

2


Introduction

One of the outcomes of instructional design (ID) must be an assessment of the impact of the

instruction design. For those that utilize the ADDIE design model this is the last step in the

process, E – Evaluation. The assessment of the impact of the ID is discussed in Section

3.0, Effectiveness of the Intervention. Section 1 – Topic, Audience and Purpose (TAP) and

Section 2, Problem will reintroduce the audience to the scope of the original User Needs

Analysis.

1.0 TAP (Topic, Audience & Purpose)

“My Client” is a National Product Service Division (NPS) that is part of a $3 billion

organization. NPS consists of 22 facilities geographically located in the continental United

States, Hawaii and Puerto Rico. These facilities will conduct more than a million repairs

annually on consumer goods, primarily electronics, for more than 3,000 company stores.

This division was formally created as part of the company in 1992.

Having created service systems for another large retailer during the mid to late

1980’s and then personally having a key role in the formation of the service division for my

client during the early 1990’s I am very familiar the activities and processes within the

service division.

The problem being faced is that over the years the processes within the service

division have not changed significantly, and there has been very little turn over in personnel

in the previous two decades. Over the years formal training within the division on their core

systems had not been necessary given their low turnover rates. With the company looking

at emerging markets the service division would like to create a formal training program on its

core system.

I have personally and professionally known the Senior Vice President of the service

division since the mid 1980’s. This will help facilitate the process since I do not have to

establish my creditability in getting the job done. Conversely, I know the SVP of the service

3


division to be very direct and goals oriented which will help to eliminate scope creep and will

insure whatever resources and time necessary, from the national product service division, to

make this a successful endeavor will be provided.

1.2 Topic Product Service Systems Orientation

1.3 Audience Attendees will be a mixed blending of work associates, supervisors and managers.

Participants will be motivated to learn, as they fully understand this training is designed to

make their work flow easier and less manual than it currently is. Each individual will be

looking for how it integrates into their specific day to day activities.

1.4 Purpose At the conclusion of training individuals should have a firm grasp of how to navigate the

system, understanding of the characteristics or functionality of the system, which section of

the applications apply to their job role and basic knowledge of how to utilize these portions of

the application. This course is not designed to make user “experts”, rather participants will

have the fundamental knowledge required to use the application as part of their normal daily

tasks.

1.5 Format The course of instruction is to be conducted as an instructor led program of learning.

Instructor Led Training (ILT) means that the course of instruction will not only be conducted

by a person, but that person will be available to monitor; review and assess; and answer any

questions or concerns throughout the course.

2.0 Problem

4


Needs will be reviewed and presented in two different sections: (1) learners and (2)

instructors.

2.1 Learners The problem being faced is that over the years the processes within the service division

have not changed significantly, and there has been very little turn over in personnel in the

previous two decades. Over the years formal training within the division on their core

systems had not been necessary given their low turnover rates. With the company looking

at emerging markets the service division would like to create a formal training program on its

core system.

2.2 Instructors The instructor(s) cannot just “teach the class”. The instructor must not only be an instructor,

but must also be a subject matter expert (SME); an individual capable of demonstrating their

knowledge of the lesson plan. The benefit of the instructor being a SME is the learning can

go beyond the scope of the objectives and goals. While in most situations it is desirable for

the ID to provide only that which the client has asked for, in this case it is desirable for the

instructor to be part of the client base. This will add a level of creditability to completion of

the goals and objectives.

2.3 Core problems with existing instructional design Fundamentally there is not a problem with the existing instructional design (ID). –The root

problem is there is NOT any existing instructional design.

2.4 Overview of existing skills, knowledge and limitations When conducting a review of the existing skills and knowledge there does not appear to be a

problem in this area. However, there is a limiting factor on a much broader scale; one that

extends beyond the individual level of the prescribed audience. This limitation has to do with

preparation for the future, and growth of the national product service division.

5


While the existing audience has not been exposed to the positive impacts of ID, they

have achieved a sufficient level of knowledge to perform their prescribed tasks.

3.0 Effectiveness of the Intervention

Effectiveness of the intervention must be measured both qualitatively and quantitatively. For

this several instruments were designed to measure the learners’ capabilities and the

effectiveness of the instructional design (ID). An end of course Training Evaluation

Questionnaire (Appendix A) was administered to each of the learners at the completion of

the course. Each of a series of questions was proposed to the learner utilizing a Likert

scale. This scale was graded 1 through 5. With 1 = “Strongly disagree”, or the lowest, most

negative impression; 3 = “Neither agree nor disagree”, or an adequate impression; and 5 =

“Strongly agree”, or the highest, most positive impression. Percentages may not sum to

100% due to rounding to the nearest percentage.

3.1 Training Content As determined by the results of the end of course Training Evaluation Questionnaire

(Appendix B) Training Content received an 89% acceptance rating at level 5 and an 7%

acceptance rating at level 4 for an overall acceptance rating of 96%. 4% of the learners

rated the Training Content at level 3 with no respondents replying at levels 1 or 2. All nine

learners responded to each of the three questions presented. This section consisted of

three post implementation questions.

3.2 Training Design As determined by the results of the end of course Training Evaluation Questionnaire

(Appendix B) Training Design received an 38% acceptance rating at level 5 and an 56%

acceptance rating at level 4 for an overall acceptance rating of 94%. 4% of the learners

rated the Training Design at level 3 with an additional 2% of the learners giving a rating at

level 2. This section consisted of five post implementation questions.

6


3.3. Training Instructor (Facilitator) The Training Instructor (Facilitator) received a 100% rating at level five as compiled from the

results of the Training Evaluation Questionnaire (Appendix B). This section consisted of two

post implementations questions.

3.4 Training Results As determined by the results of the end of course Training Evaluation Questionnaire

(Appendix B) Training Results received an 89% acceptance rating at level 5 and an 11%

acceptance rating at level 4 for an overall acceptance rating of 100%. This section consisted

of two post implementation questions.

3.5 Summary Overall 69% of the learners rated the intervention at a level 5 with 27% responding at level 4.

This resulted in an overall acceptable rating of 96% with 3% responding at level 3 and 1%

responding unfavourably at level 2. No learners indicated N/A on their answers.

While the administration and interpretation of an end of course evaluation can be

used to determine the overall impact of an intervention it is usually acceptable to receive an

early indicator of an acceptable level of performance.

It should be noted that under Kirkpatrick’s Four Level Training Evaluation Model

(Appendix C) that only the requirements for a level one evaluation were completed.

However, the conclusion for level one is the ID was effective and adequately supported the

client’s needs.

Additionally, it should be noted that as part of the post implementation assessment

several forms of assessment materials were utilized to determine a level of understanding

and competencies from the learners. These instruments were not utilized as part of the

impact assessment. Instruments of this type are best utilized as part of a level two

evaluation. As part of the level two evaluation the results from these instruments should be

compared to the same instruments being given to the learners at a prescribed interval, i.e.,

7


30 days, 60 days, etc. Normally, level two would be completed in the form of pre and post-

tests. The purpose of level two is to measure “Did the learners learn anything?”. This will

allow for the assessment or impact measurement at level two.

8


Appendix A. Training Evaluation Questionnaire

Training Evaluation Questionnaire

Course Name: _________________________________ Training Location: _________________________________ Participant Name (optional): _________________________ Date: ________________ Job Title: ________________________________________ Years in present position? <1 1-3 3-5 5+ INSTRUCTIONS Please circle your response to the items. Rate aspects of the Training on a 1 to 5 scale: 1 = “Strongly disagree, “or the lowest, most negative impression 3 = “Neither agree nor disagree, “or an adequate impression 5 = “Strongly agree,” or the highest, most positive impression Choose N/A if the item is not appropriate or not applicable to this Training. Your feedback is sincerely appreciated. Thank you.

TRAINING CONTENT (circle your response to each item.)

1. I was well informed about the objectives of this training.

1 2 3 4 5 N/A

2. This training lived up to my expectations.

1 2 3 4 5 N/A

3. The content is relevant. 1 2 3 4 5 N/A TRAINING DESIGN (circle your response to each item.) 4. The training objectives were clear to

me. 1 2 3 4 5 N/A

5. The training activities stimulated my learning.

1 2 3 4 5 N/A

9


Appendix A. Training Evaluation Questionnaire (cont)

TRAINING DESIGN (cont) (circle your response to each item.) 6. The activities in this training gave me

sufficient practice and feedback. 1 2 3 4 5 N/A

7. The difficulty level of this training was appropriate.

1 2 3 4 5 N/A

8. The pace of this training was appropriate.

1 2 3 4 5 N/A

TRAINING INSTRUCTOR (FACILITATOR) (circle your response to each item.) 9. The instructor was well prepared. 1 2 3 4 5 N/A 10. The instructor was helpful. 1 2 3 4 5 N/A

TRAINING RESULTS (circle your response to each item.) 11. I accomplished the objectives of this

training. 1 2 3 4 5 N/A

12. I will be able to use what I learned in this training.

1 2 3 4 5 N/A

COMMENTS: ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

10


Appendix B. Training Evaluation Questionnaire Results

TRAINING CONTENT Likert Scale

1 2 3 4 5 N/A

1. I was well informed about the objectives of this training. 1 1 7

2. This training lived up to my expectations. 1 8

3. The content is relevant. 9

RAW SCORES 0 0 1 2 24 0

RAW PERCENTAGES 0% 0% 4% 7% 89% 0%

TRAINING DESIGN Likert Scale

1 2 3 4 5 N/A

1. The training objectives were clear to me.

1 8

2. The training activities stimulated my learning.

1 1 1 6

3. The activities in this training gave me sufficient practice and feedback.

7 2

4. The difficulty level of this training was appropriate.

1 8

5. The pace of this training was appropriate.

8 1

RAW SCORES 0 1 2 25 17 0


TRAINING INSTRUCTOR (FACILITATOR) Likert Scale

1 2 3 4 5 N/A

1. The instructor was well prepared. 9

2. The instructor was helpful. 9

RAW SCORES 0 0 0 0 18 0


TRAINING RESULTS Likert Scale

1 2 3 4 5 N/A

1. I accomplished the objectives of this training.

1 8

2. I will be able to use what I learned in this training.

1 8

RAW SCORES 0 0 0 2 16 0


TOTAL RESULTS Likert Scale

1 2 3 4 5 N/A

RAW SCORES 0 1 3 29 75 0


11


Appendix C. Kirkpatrick’s Four Levels of Evaluation

The four-levels of evaluation consist of (Kirkpatrick, 1994): 1. Reaction – how learners react to the learning process

Evaluation at this level measures how the learners react to the training. This level is

often measured with attitude questionnaires that are passed out after most training

classes. This level measures one thing: the learner’s perception (reaction) of the

course).

2. Learning – the extent to which the learners gain knowledge and skills

This is the extent to which participants change attitudes, improve knowledge, and

increase skill as a result of participating in the learning process. It addresses the

question: Did the participants learn anything? The learning evaluation requires some

type of post-testing to ascertain what skills were learned during the training. In addition,

the post-testing is only valid when combined with pre-testing, so that you can

differentiate between what they already knew prior to training and what they actually

learned during the training program.

3. Behavior – capability to perform the learned skills while on the job

This evaluation involves testing the students capabilities to perform learned skills while

on the job, rather than in the classroom. Level three evaluations can be performed

formally (testing) or informally (observation). It determines if the correct performance is

now occurring by answering the question, “Do people use their newly acquired learning

on the job?”

4. Results – includes such items as monetary, efficiency, moral, etc.

This is the final results that occur. It measures the training program's effectiveness, that

is, “What impact has the training achieved?” These impacts can include such items as

monetary, efficiency, moral, teamwork, etc.

12


(www.nwlink.com/~donclark/hrd/ahold/isd.html)

Figure 1. Instructional System Design Concept Map

13


References

Kirkpatrick, D. L. (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.

Documents

Effectiveness of the Intervention Product Service Systems ... · 3.0, Effectiveness of the Intervention. Section 1 – Topic, Audience and Purpose (TAP) and Section 2, Problem will