Embed Size (px)
DESCRIPTION
In this study, Analysis of Patterns in Time (APT) is used to analyze gameplay to provide evidence of a learner’s understanding of concepts modeled in a game. Gameplay data form an APT map of joint and sequential patterns. An algorithm compares these patterns with patterns based on optimal strategies derived from the game’s conceptual model. We discuss the results of using APT for analysis of game sessions of the online Diffusion Simulation Game. Ted Frick and I presented this at the November, 2014 AECT conference in Jacksonville, FL.
Citation preview
USING PATTERN
MATCHING TO
ASSESS
GAMEPLAY
RODNEY D. MYERS, PH.D.
INDEPENDENT SCHOLAR
THEODORE W. FRICK, PH.D.
PROFESSOR EMERITUS, INDIANA UNIVERSITY
1
INTRODUCTION
Using Analysis of Patterns in Time (APT) to measure and
analyze a learner’s interactions with a serious game.
• Overview of MAPSAT and APT
• Comparison with traditional methods
• Examples and explanations
• Case Study
• The Diffusion Simulation Game and diffusion of
innovations theory
• Using APT to analyze gameplay data
• Using APT for Formative Assessment
• Concluding Remarks
2
OVERVIEW OF MAPSAT
Map & Analyze Patterns & Structures Across Time: 2 methods
• Analysis of Patterns in Time (APT)
• Analysis of Patterns in Configuration (APC)
APT
• Different approach to measurement and analysis
• Create a temporal map which characterizes temporal events
• Look for temporal patterns within a map
• Count them (event pattern frequency)
• Estimate likelihood (relative frequency)
• Aggregate time (event pattern duration)
• Estimate proportion time (relative pattern duration)
3
HOW IS APT
DIFFERENT?
• Traditional quantitative methods of measurement and
analysis
• Obtain separate measures of variables for each case
• Statistically analyze relations among measures
• We relate measures
Example of spreadsheet data:
4
HOW IS APT
DIFFERENT?
• Analysis of Patterns in Time
• Create temporal map for each case
• Query temporal map for patterns
• We measure relations directly
Example of spreadsheet data:
5
Map Query 1 Query 2
1 0.30 0.58
2 0.25 0.67
3 0.40 0.56
WHAT IS A TEMPORAL
MAP?
6
Example of temporal map of weather
JTE Unix Epoch
Time Started:
Duration of
JTE
Season
of Year
Air
Temperature
(degrees F)
Barometric
Pressure
(p.s.i.)
Precipitation Cloud
Structure
1 1417436508:
dur. = 1470
{ Fall { 33 { Above 30 { Null { Cirrus
2 1417437978:
dur. = 2277
| | { Below 30 | |
3 1417440255:
dur. = 2554
| | | | { Nimbus
Stratus
4 1417442809:
dur. = 794
| | | { Rain |
5 1417443603:
dur. = 1095
| { 32 | | |
6 1417444698:
dur. = 477
| | | { Sleet |
CODEBOOK FOR
OBSERVING WEATHER
EVENTS
Classification 0 Name: Season of Year
Classification Value Type = Nominal
Number of categories (temporal event values) = 5
Category 0 = Null
Category 1 = Fall
Category 2 = Winter
Category 3 = Spring
Category 4 = Summer
Classification 1 Name: Air Temperature
Classification Value Type = Interval
Units of measure = degrees Fahrenheit
7
CODEBOOK FOR
OBSERVING WEATHER
EVENTS
Classification 2 Name: Barometric Pressure
Classification Value Type = Ordinal
Number of categories (temporal event values) = 3
Category 0 = Null
Category 1 = Above 30 psi
Category 2 = Below 30 psi
Classification 3 Name: Precipitation
Classification Value Type = Nominal
Number of categories (temporal event values) = 4
Category 0 = Null
Category 1 = Rain
Category 2 = Sleet
Category 3 = Snow
8
QUERY A TEMPORAL
MAP: EXAMPLE
Query 1. Here is a 2-phrase APT Query:
WHILE the FIRST Joint Temporal Event is true (Phrase 1):
Season of Year is in state starting or continuing, value = FallBarometric Pressure is in state starting or continuing, value = Below 30Cloud Structure is in state starting or continuing, value = Nimbus Stratus
• Duration when Phrase 1 is True = 13,436 seconds (out of 19,584 seconds total). Proportion of Time = 0.68607
• Joint Event Frequency when Phrase 1 is True = 12 (out of 18 total joint temporal events). Proportion of JTEs = 0.66667
THEN while the NEXT Joint Temporal Event is true (Phrase 2):
Season of Year is in state starting or continuing, value = FallBarometric Pressure is in state starting or continuing, value = Below 30Precipitation is in state starting or continuing, value = RainCloud Structure is in state starting or continuing, value = Nimbus Stratus
• Duration when Phrase 2 is True = 4,086 seconds (out of 19,584 seconds total), given all prior phrases are true. Proportion of Time = 0.20864
• Joint Event Frequency when Phrase 2 is True = 3 (out of 18 total joint temporal events), given all prior phrases are true. Proportion of JTEs = 0.16667
• Conditional joint event duration when Phrase 2 is true, given all prior phrases are true = 0.30411 (4,086 out of 13,436 seconds (time units).
• Conditional joint event frequency when Phrase 2 is true, given all prior phrases are true = 0.25000 (3 out of 12 joint temporal events).
9
RESULT OF QUERY FOR
APT PATTERN IN
TEMPORAL MAP
10
Map Query 1 Query 2
1 0.30 0.58
2 0.25 0.67
3 0.40 0.56
The conditional joint event duration of the 2-phrase
pattern specified in Query 1 becomes the measure
that is entered into the spreadsheet
Thus, the variable is the pattern specified Query 1
and its value is 0.30.
DEMO OF APT QUERIES
ON WEATHER PATTERNS
If we have a good Internet connection:
https://www.indiana.edu/~simed/aptdemo/aptdsg.php
11
EXAMPLE OF APT TEMPORAL
MAP FOR THE DIFFUSION
SIMULATION GAME
If we have a good Internet connection:
https://www.indiana.edu/~simed/aptmultimap/aptdsg.php
12
THE DIFFUSION SIMULATION
GAME & DOI THEORY
13
Case Study: Using APT for Serious Games Analytics
Diffusion Simulation Game (DSG)
USING APT TO ANALYZE
GAMEPLAY DATA
Generalizations from
DOI theory
14
DSG activities
Adopter types
Decision phases
Example:
Mass media should be
effective in spreading
knowledge about an
innovation, especially
among innovators and
early adopters
Local Mass Media & Print
Innovators & Early Adopters
Awareness & Interest
9 strategies: patterns of joint occurrences
file:///Users/tedfrick/Documents/AECT%202014/View%20Temporal%20Map%
20DSG%20MultiMap.html
USING APT TO ANALYZE
GAMEPLAY DATA
• Revised DSG to require login to track changes in
gameplay over time
• Reviewed “finished” games
• 109 players finished 1 or more games
• 27 players finished 2 or more games
• 14 players finished 3 or more games
• Selected 3 players to serve as examples
15
GAME OUTCOMES
16
Game Outcome Adoption Points
Maximally Successful 220
Highly Successful 166 – 219
Moderately Successful 146 – 165
Unsuccessful 0 - 145
Game Outcomes
Player 1 Un Un Un Md
Player 2 Un Un Md Hi Md Hi Md Un Hi Hi Hi
Player 3 Un Hi Md Hi Mx Hi
EXAMPLE QUERY
17
Query Result for Player 1, Game 3
WHILE the FIRST Joint Temporal Event is true (Phrase 1):
Diffusion Activity is in state starting or continuing, value = Local Mass
Media
Turn Rank is in state starting or continuing, value <= 3
• Duration when Phrase 1 is True = 1 moves (out of 59 DSG moves
total). Proportion of Time = 0.02222
• Joint Event Frequency when Phrase 1 is True = 1 (out of 74 total joint
temporal events). Proportion of JTEs = 0.01351
EXAMPLE QUERY
18
Player 1 Un Un Un Md
Overall 0.00 0.00 0.03 0.03
High 0.00 0.00 0.02 0.03
Low 0.00 0.00 0.02 0.00
Player 2 Un Un Md Hi Md Hi Md Un Hi Hi Hi
Overall 0.00 0.05 0.03 0.07 0.05 0.06 0.04 0.05 0.02 0.10 0.10
High 0.00 0.00 0.00 0.02 0.02 0.02 0.02 0.00 0.00 0.05 0.05
Low 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.02 0.00 0.02 0.02
Player 3 Un Hi Md Hi Mx Hi
Overall 0.02 0.07 0.10 0.10 0.05 0.09
High 0.02 0.02 0.03 0.03 0.03 0.03
Low 0.00 0.05 0.05 0.08 0.03 0.06
Use of Local Mass Media activity by game outcome and strategy
rank for turn.
USING APT FOR FORMATIVE
ASSESSMENT DURING GAMEPLAY
19
• Summative
• Used by instructor and/or learner
• Evidence of understanding and application
• Formative
• Dynamic analysis of gameplay
• Provide scaffolds (e.g., hints, coaching)
• Requested by learner
• Before turn: hint
• After turn: explanation or prompt for reflection
• Analyze prior gameplay maps
• Identify persistent misconceptions
USING APT FOR FORMATIVE
ASSESSMENT DURING GAMEPLAY
20
Generalization 5-13: Mass media channels are relatively more
important at the knowledge stage, and interpersonal channels
are relatively more important at the persuasion stage in the
innovation-decision process (p. 205).
Generalization 7-22: Earlier adopters have greater exposure to
mass media communication channels than do later adopters (p.
291).
Player 3 Un Hi Md Hi Mx Hi
Overall 0.02 0.07 0.10 0.10 0.05 0.09
High 0.02 0.02 0.03 0.03 0.03 0.03
Low 0.00 0.05 0.05 0.08 0.03 0.06
Poor use of
mass media
CONCLUDING REMARKS
21
“Using Pattern Matching to Assess Gameplay”
to be published in:
Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.). (2015). Serious
game analytics: Methodologies for performance
measurement, assessment, and improvement. New York, NY:
Springer.
Contact us:
Rod Myers – [email protected]
Ted Frick – [email protected]
