Upload
jeffreydavidscottzo
View
334
Download
0
Tags:
Embed Size (px)
Citation preview
2. CONTEXT| Job Map
3. Context| DRIVERS
Introduce multimedia assessment method
Increase consistency between end-of-training assessments and
on-the-job evaluations
Increase authenticity of assessment
Sam Gonzales, Director of Human Performance Technology
4. Context | DRIVERS
Current Assessment
Pencil-and-paper
Question types
Multiple choice
Short answer
Time consuming to score
New Assessment
Computer administered
Multiple-choice
Automatically scored
8 video clips
Identify task
Errors made?
Identify errors or critical aspect
5. Context| Action taken
6. Context| Results of the test
FREQUENT
UNUSUAL
FREQUENT
UNUSUAL
7. Context| Results of the test
8. Context| Results of the test
Correlation between end-of-training assessment and on-the-job
performance evaluation was significantly improved in the
experimental group
9. Key Stakeholders | Trainees
Concerned about the difficulty and fairness of video
questions
10. Key Stakeholders | Instructors
Concerned about poor end-of-training performance reflecting on
their own performance
11. Key Stakeholders | Supervisors
Concerned about the changing workforce and preparedness of new
employees
12. Performance Issues
Facilitating preflight checks when unusual situations occur
Unusual situations preflight
Skills for facilitating preflight checks
13. Performance Issues
Dealing with difficult passengers when unusual situations
occur
Difficult passengers increase task load and communication
demand
Communication skills for passenger interaction
14. Context| Results of the test
FREQUENT
UNUSUAL
FREQUENT
UNUSUAL
15. Solution | Eliminate confounding factor
FREQUENT
UNUSUAL
FREQUENT
UNUSUAL
16. Solution | Add New Performance Goals
Redesign the course, assessment and evaluation
Add learning content to include instruction, discussion and
additional practice handling unusual situations
Add traditional test questions to assess the learners understanding
of how to complete tasks in unusual situations
Prompt the evaluator to distinguish between frequent vs. unusual
situations when completing observational evaluations of
performance
17. Evaluation | Conduct a second study
Conduct a second study to evaluate the effectiveness of the added
performance goals
Control group of 20 completes original course and assessment, but
the redesigned evaluation
Experimental group of 20 is trained, assessed and evaluated using
all the redesigned tools
18. Evaluation | Analyze Results
Conduct reliability analysis for both training assessments without
distinction between frequent and unusual performance goals
Conduct a second reliability analysis on the experimental results
WITH distinction between frequent and unusual performance
goals
19. Evaluation | Analyze Results
For each group, find correlation between end of training assessment
and on the job performance evaluations without distinguishing
between frequent and unusual performance goals
20. Evaluation | Analyze Results
Compare on the job performance evaluations between the two groups
WITH distinction between frequent and unusual performance
goals
21. Summary | Recommendations
Add content, assessment, and performance evaluation to support
goals of completing tasks in unusual situations
Adopt multimedia test method
22. References
Ertmer, P. A. & Quinn, J. (2007.) The ID casebook: Case studies
in instructional design, 3rd ed. Upper Saddle River, NJ: Pearson
Prentice Hall.
Fraenkel, J. R. & Wallen, N. E. (2008.) How to design and
evaluate research in education.7th ed.New York:McGraw Hill.
Hale, J.(2007.)The performance consultants fieldbook:Tools and
techniques for improving organizations and people, 2nd ed.San
Francisco:Pfeiffer.