Upload
august
View
30
Download
1
Tags:
Embed Size (px)
DESCRIPTION
What to Consider When Developing and Evaluating Games for Training . Teresita Marie Sotomayor, PhD Science and Technology Manager US Army Research Laboratory Simulation and Training Technology Center. Outline. This presentation will explore: - PowerPoint PPT Presentation
Citation preview
U.S. Army Research, Development and Engineering Command
*Unclassified – Approved for Public Release*
What to Consider
When Developing and Evaluating Games
for Training
Teresita Marie Sotomayor, PhDScience and Technology ManagerUS Army Research Laboratory Simulation and Training Technology Center
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Outline
• This presentation will explore:
– Generational differences and the characteristics of the new generation of trainees
– Features of games that motivate and engage individuals
– An overview of the elements that need to be considered when evaluating the effectiveness of games in a training environment
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Generational Differences• When developing games for training it is important to
consider nested generational groups within the learning population.
Generation X
• Prefer fast-paced
presentation of
information
• Frequent feedback
• Sense of
accomplishment
• Visual – grown up in
the 3Dworld
• Single channelers
Generation Z
• Born with complete
technology
• Highly creative and
collaborative
• Multi-taskers
• React fast
Generation Y
• grown up with the
internet at their
fingertips
• World to them is virtual
• Possibilities are
endless
• Multi-channelers
Training Environment
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Why?
• When designing tools for training it is important to understand targeted audience:
– Characteristics
– Learning Preferences
– Cognitive Styles
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Sample Population Profile
Educational background Active Duty Combat Medics and Reservists: some college
Prior level of medical training Active Duty and Reservist: some prior and combat medical training
Computer literacy level Active Duty: intermediate level computer literacy
Reservists: basic computer literacy
Male/female ratio 5 to 1 male to female
Age range 18 to 50
level 12th grade
Learning style Visual and Kinesthetic
Background Various backgrounds
Some foreign studentsMotivation for completing the training
Job requirement, advancement, job knowledge, personal pride
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Sample Population Profile
AgeRange: 18 to 42Generation X: 13%Generation Y: 87%
GenderMale: 73%Female: 27%Male to Female Ratio: 3 to1
ExperienceCombat Operations: 8%Combat Medic: 2%Civilian Setting: 29%
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Sample Population Profile
Video Game Experience: 93 %Ability: Beginner: 29% Intermediate: 48% Expert: 23%
Computer Ability: Beginner: 11% Intermediate: 72% Expert: 16%
EducationHS: 30 %Some College: 53%Associate: 10%Bachelor: 6%Post Bachelor: 1%
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Age Distribution
0
10
20
30
40
50
60
70
80
18-21 22-25 26-29 30-33 34-37 38-42
Age Subsets
Num
ber
of M
edic
sNumber of Medics
Gender Generation X
Total # (Percentage)
Generation Y
Total # (Percentage)
Male 17 (12.9) 115 (87.1)
Female 6 (12.5) 42 (87.5)
Total 23 (12.7) 157 (87.3)
93%
Sample Population Profile
Gender and Generation Categorization
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Video Game Experience
ExperienceNo Experience
93%
Sample Population Profile
Id # Age Generation Gender Education
10 26 Y Female HS
18 30 Y Female Bachelors
1 41 X Male Bachelors
26 22 Y Male Some College
8 40 X Male Associate
19 18 Y Female HS
47 19 Y Male HS
115 20 Y Female Some College
148 19 Y Female HS
138 27 Y Female Bachelors
75 23 Y Female Some College
68 26 Y Female Associate
126 35 X Male Some College
Descriptive Characteristics of Subjects
without Video Game Experience
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
First Person Video Game Ability
29%
48%
23%
BeginnerIntermediateExpert
Sample Population Profile
GenderFirst Person Video
Game Ability
Beginner Intermediate Expert
Computer Ability
Beginner Intermediate Expert
Female 32(67%) 15(31%) 1(2%) 6(13%) 38(81%) 3(6%)
Male 21(16%) 71(54%) 40(30%) 14(11%) 91(69%) 26(20%)
First Person Video Game and Computer Ability Self Assessment
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Distribution of Hours Spent Gaming
0
20
40
60
80
100
120
0-5 `6-11 `12-17 18-23 24-29 30-35 36-41 42-47 48-50
Number of Hours
Num
ber o
f Med
ics
Number of Medics
Sample Population Profile
Distribution of Video Game Hours on a Typical Week
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Games are a form of fun
• Games are a form of play
• Games have rules
• Games have goals
• Games have outcome and feedback
Why Games Engage?
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Games are adaptive
• Games have win states
• Games have conflict/competition/challenge/opposition
• Games have problem solving
• Games are interactive
• Games are immersive
Why Games Engage?
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Games are Interactive
• Interactivity in Games reinforces:
- Individual Control
- Trial-and-Error
- Constant Change
Why Games Engage?
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Instructional Games• Instructional Games:
– Designed or modified to meet training objectives
– Includes rules, constraints, and activities that closely replicate the constraints of the real world
tasks being trained
– Must be incorporated in the POI in a manner that ensures that trainees understand the training
objectives and receive feedback on performance against the training objectives
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Instructional Games• Instructional Games:
– Should be considered as a training aid tool
– If developed as a stand-alone training tool, the game must be designed such that all instructional
capabilities provided by the trainer are incorporated
– Feedback should be incorporated as a structured guided activity such that learning can be
fostered and the experience can be meaningful to training
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Instructional Games
• Four main aspects of a “serious game”:
1. Context of the game and usage is essential to the effectiveness of the game in the training
environment
2. Learner Specification – user profile to understand characteristics, learning preferences and
cognitive styles of the targeted group of learners
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Instructional Games
• Four main aspects of a “serious game”:
3. Representation – how levels of immersion, fidelity, and interactivity are integrated such that it is
effective and provides the required level of engagement
4. Pedagogical model – learning theory and approaches behind the application to ensure that it is
used effectively to support specified learning
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
When Designing a “Serious Game”
• USER Centric Approach
- Gap Analysis and Requirement Definition
• Structured SME Discussions
• Observation of training to ID objectives and gaps in training
• Training Curriculum Review
• Task Analysis
- Refinement
• Usability Studies
- Assessment
• Training Effectiveness Evaluation
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
The usability of an interactive system can be defined, in part, in terms of
how easily a user can access and use the intended functionality of the
system to meet task objectives.
Usability Studies
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Interactive systems should be designed such that they are intuitive,
effective, and subjectively acceptable to users.
• Performing a usability study early on can provide valuable information for
system development.
Usability Studies
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Early identification of user issues translates into reduced system redesign
costs, enhanced user satisfaction and decreased user training time and the
need for technical support.
Usability Studies
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• User Centric Methodology utilized to:
- Gather user (instructors and potential users) feedback regarding the current state of the
technology being developed
- Explore the feasibility of incorporating innovative training tools in the current program of
instruction
Usability Studies
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Observation of trainees and trainers interacting with the system
• Feedback on their reactions captured through surveys and participation in
structured focus group interviews.
Methodology
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Questionnaires are effective instruments to collect:
– Demographic data
– Feedback regarding their experience with the Simulation
– Reaction to training
Methodology
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Usability questionnaires are effective instruments to gather subjective data
regarding user’s reaction:
- Benefit to Training, System Usability, Presence and Realism, and Motivation to Use
- Training Objectives, Knowledge and Skills, Overall Reaction to Training
Methodology
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Focus groups are effective instruments to gather collective subjective data that
can’t be collected via survey
- Feedback about the current state of the prototype
- Scenario/Environment, Techniques, User Interface, Training Objectives and After Action Review
(AAR)
Methodology
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Training Effectiveness Evaluation
• The main purpose of a training program is to impart the acquisition of knowledge, skills, and competencies.
Evaluating a particular training is a challenging task
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Training Effectiveness Evaluation
• Developed by Donald Kirkpatrick in 1952 and published in 1959 in the Journal of the American Society of Training Directors
• Provides a methodology for evaluating training programs
• It is widely used because is simple and practical and can be applied to different situations (is flexible)
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level I
Reaction Criteria
Reaction: Evaluates user affective and attitudinal response to the training program.
Focuses on self-report measures.
Instruments: Survey, questionnaires, focus groups, etc.
Level II
Learning Criteria
Knowledge Acquisition: Evaluates user’s increase in knowledge and capability as a result of the
training program.
Focuses on learning outcomes.
Instruments: pre-/post Knowledge Test, Interview or on-the-job assessments, etc.
Level III
Behavior Criteria
Skill Transfer: Evaluates changes in behavior on the job as a result of the training program.
Focuses on the job performance measures.
Instruments: self-assessments, on-the-job performance ratings, etc.
Level IV
Results Criteria
Benefits: Evaluates the effect on the organization resulting from the training program.
Focuses on long-term productivity performance measures.
Instruments: Utility Analysis Instruments, Organizational performance reports, QA Reports, etc.
Kirkpatrick's Model
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 1 Reaction
• Measure of how participants feel about the various aspects of a training program.
• Why should we measure reaction?
- People must like a training program to obtain the most benefit.
- To ensure participants are motivated and interested in learning.
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 1 Reaction
• Guidelines:- Determine what needs to be found out about the training
- Develop Instruments: - Questionnaires, focus groups, researcher observations on
participants reaction to the simulation
- Need to ensure that reactions can be tabulated and quantified
- Obtain honest reactions by making the sheets anonymous
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 2 Learning
• This is a measure of the knowledge acquired, skills improved, or attitude changed due to training. In our context we are aiming to improve knowledge of concepts, principles, or techniques.
• Why?- Measure of reaction of trainees is not an indication that
learning has taken place.
- Will indicate what principles, facts, and techniques were understood and absorbed by trainees.
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 2 Learning
• Guidelines:
- Use a Control Group
- Develop instruments to evaluate knowledge before and after training:• Pre/post test, assessments, ratings of performance
- Use 100 percent scale
- Evaluate results statistically so that learning can be proven
- Use the results of the evaluation to take appropriate action
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 3 Behavior
• This is a measure of the extent to which participants change their on-the-job behavior because of training. It is also referred as transfer of training.
• Why?
- There is a big difference between knowing principles and techniques and properly using them on the job.
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 3 Behavior
• Guidelines:
- Use a Control Group
- Allow enough time for change in behavior to take place
- Develop quantitative instruments:• Scenario based simulations, self efficacy instruments, on-the-job
observations
- Survey or interview one or more of the following: trainees, instructors, superiors, etc
- Use the results of the evaluation to take appropriate action
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
• Measure of the long term results of the training program in terms of reduced costs,
higher quality, higher productivity, lower rates of employees turnover or absenteeism,
etc.
• Why?
It can be used to support acquisition of a specific training device or to include methodology in POI.
Warning: Evaluations become more difficult, complicated and expensive. How much of the
improvements shown over time are due to training and not other variables.
Level 4 Results
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
Level 4 Results
• Guidelines:
- Use a Control Group
- Allow enough time for results to be achieved
- May be obtained from unit performance reports, quality inspections and interviews with Senior people
- Use the results of the evaluation to take appropriate action
SFC Paul Ray Smith Simulation & Training Technology Center
Unclassified – Approved for Public Release*
?? QUESTIONS ??