Serious Play Conference Los Angeles, CA July 22, 2014 Girlie C.
Delacruz Games as Assessment: Validating Patterns of Gameplay Via
Cluster Analysis
Slide 2
ENGAGE Team Regents of the University of California Eva Baker
Principal Investigator Girlie Delacruz Project Director Greg Chung
Project Director Noelle Griffin Associate Director Ayesha Madni
Senior Researcher Markus Iseli Senior Researcher Alan Koenig Senior
Researcher
Slide 3
Typical Metrics
Slide 4
Determine Proficiency
Slide 5
Diagnostic Assessment
Slide 6
Games As Formative Assessment Formative Assessment: Use and
interpretation of task performance information with intent to adapt
learning, such as provide feedback. (Baker, 1974; Scriven, 1967).
Formative Assessment: Use and interpretation of task performance
information with intent to adapt learning, such as provide
feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use
and interpretation of task performance information with intent to
adapt learning, such as provide feedback. (Baker, 1974; Scriven,
1967). Formative Assessment: Use and interpretation of task
performance information with intent to adapt learning, such as
provide feedback. (Baker, 1974; Scriven, 1967).
Slide 7
Games As Formative Assessment Games as Formative Assessment:
Use and interpretation of game performance information with intent
to adapt learning, such as provide feedback. (Baker, 1974; Scriven,
1967). Games as Formative Assessment: Use and interpretation of
game performance information with intent to adapt learning, such as
provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative
Assessment: Use and interpretation of game performance information
with intent to adapt learning, such as provide feedback. (Baker,
1974; Scriven, 1967). Games as Formative Assessment: Use and
interpretation of game performance information with intent to adapt
learning, such as provide feedback. (Baker, 1974; Scriven,
1967).
Slide 8
Qualitative Metrics
Slide 9
Slide 10
ENGAGE
Slide 11
Project Goals
Slide 12
ENGAGE Games
Slide 13
Go Vector Go
Slide 14
Current Study: Validating Patterns of Gameplay
Slide 15
Research Questions 1.What patterns of gameplay are identified
using cluster analysis? 2.What is the association between
identified clusters and performance on the related item
administered the pretest?
Slide 16
Force and Motion: Constant Velocity Big Idea A change in the
position and motion of an object is a result of the manipulation of
forces acting on an object. Example Learning Outcome Manipulate
friction, mass, force magnitude, force direction, and slope to
cause an object to move at constant velocity.
Slide 17
Pretest Item
Slide 18
Diagnostic Options 1. No changes in forces. 2. Vector know what
to do now. 3. A lot of forces. 4. A lot of energy. Prompt Why does
Vectors speed not change?
Slide 19
Cluster Analysis
Slide 20
Validity Evidence for Gameplay Patterns
Slide 21
Choosing Clustering Variables LevelVector isNeed toBy Initial
Force Magnitude Initial Surface Friction 4 Not moving Inc. speed
Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed
Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed
Dec. friction 1 (Reverse Thruster) 1 (Dirt)
Slide 22
Results What patterns of behavior are identified using cluster
analysis? Gameplay pattern of friction values chosen on first
attempt (correct friction value should be 0) Number of students in
cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18
Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster
178--00 Cluster 246--03 Cluster 363--33
Slide 23
Results What is the association between identified clusters and
performance on the related item administered the pretest? Gameplay
pattern of friction values chosen on first attempt (correct
friction value should be 0) Relation between gameplay pattern and
pretest item performance Number of students in cluster Level 4
Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership
did not have any significant relation to student performance on the
assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10
and 18 Cluster membership was a significant predictor of student
performance on the assessment item. Cluster 178--00 Students in
this cluster were over 2 times more likely to get the item correct
than students in Cluster 3. Cluster 246--03 Cluster 363--33
Students in this cluster were over 3 times more like to choose the
incorrect answer option Vector knows what to do now.
Slide 24
Results What is the association between identified clusters and
performance on the related item administered the pretest? Gameplay
pattern of friction values chosen on first attempt (correct
friction value should be 0) Relation between gameplay pattern and
pretest item performance Number of students in cluster Level 4
Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership
did not have any significant relation to student performance on the
assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10
and 18 Cluster membership was a significant predictor of student
performance on the assessment item. Cluster 178--00 Students in
this cluster were over 2 times more likely to get the item correct
than students in Cluster 3. Cluster 246--03 Cluster 363--33
Students in this cluster were over 3 times more like to choose the
incorrect answer option Vector knows what to do now.
Slide 25
Results What is the association between identified clusters and
performance on the related item administered the pretest? Gameplay
pattern of friction values chosen on first attempt (correct
friction value should be 0) Relation between gameplay pattern and
pretest item performance Number of students in cluster Level 4
Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership
did not have any significant relation to student performance on the
assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10
and 18 Cluster membership was a significant predictor of student
performance on the assessment item. Cluster 178--00 Students in
this cluster were over 2 times more likely to get the item correct
than students in Cluster 3. Cluster 246--03 Cluster 363--33
Students in this cluster were over 3 times more like to choose the
incorrect answer option Vector knows what to do now.
Slide 26
What Does This Mean? LevelVector isNeed toBy Initial Force
Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec.
friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec.
friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec.
friction 1 (Reverse Thruster) 1 (Dirt)
Slide 27
Implications: Validity Evidence for Gameplay Patterns
Slide 28
Next Steps
Slide 29
THANK YOU!!! Regents of the University of California
[email protected] www.cresst.org
Slide 30
Gameplay Patterns: Levels 4 - 10 - 18
Slide 31
Gameplay Patterns: Levels 10 - 18
Slide 32
Gameplay Patterns: Levels 4 - 10 - 18
Slide 33
Gameplay Patterns: Levels 10 - 18
Slide 34
Association of Cluster Membership to Pretest Performance
Slide 35
Association of Cluster Membership to Incorrect Answer Options
Chosen 2. Vector know what to do now.