Ryan Carlson - Developing Number Sense with Play: Lessons in Data Analysis

Developing Number Sense with Play: Lessons in Data AnalysisRyan Carlson

Overview• New iPad game to practice & develop

number sense

• In-school user study & pilot study addressing non-cognitive factors

• Discussion of instrumentation & evaluation of our game

Outline• Some Definitions• NumberACEr• Designing a configurable game• Instrumentation• User Study• Pilot Study• Data Analysis• Lessons Learned

Who Are We?• Founded 1998• Develop K-12 mathematics blended

curricula• 600k students / year across >2,000 schools• Cognitive Tutor

• Model student knowledge• Adaptively present new material

What is Number Sense?

“fluidity and flexibility with numbers, the sense of what numbers

mean and ability to perform mental mathematics to look at the world and

make comparisons”(Gersten & Chard, 1999)

• Foundation for further mathematics• Early number sense predicts later

mathematics ability (Geary et. al., 2013)





Personalization

CognitiveWhat you know

(positive results)

• Knowledge Tracing

Non-CognitiveEverything Else

(less well understood)

• Self-Efficacy• Grit• Learning

Orientation

NumerACEr

NumerACEr

NumerACEr

Configurability• Content• Game Parameters• Level Layout• Scripting System

• Dynamic runtime engine• React to game events• Add text / images

For non-programmers

Configurability

Configurability

Instrumentation

Boring But Important• Generate usage data & plan analyses

before deployment• Logging infrastructure & API

• Developers annotate code• Log messages sent to central storage

• Log everything!• Level started?• Speed changes?• iPad tilt?• Device suspend?

User Study• December 2014 in Raleigh, West Virginia

• 6th graders (36)• 8th graders (7)

• Played for ~10 min• Exit feedback (verbal + written)

User Study

User Study

Aggressive

Neutral(High Accuracy)

Neutral(Low Accuracy)

Cautious

User Study

Aggressive

Neutral(High Accuracy)

Neutral(Low Accuracy)

Cautious

Pilot Study• May 2015 in Savannah, Georgia

• 6th graders (200)

• Remote Deployment• 15 minute play requested• Non-Cognitive survey administered prior to

play• Self-efficacy• Grit• Mastery learning orientation

Pilot Study: Interventions

Self-Efficacy• Low SE Motivation


Grit• High Grit Increased Difficulty Earning

Brains

Learning Orientation• Low Mastery Coach Towards Mastery



Game Strategy• Aggressive Slowdown

• Reduce maximum speed

• Cautious Speedup• Increase minimum speed

• Neutral (low acc.) Instruction• Provide in-game feedback on mistakes



Pilot Study: Game Usage



Pilot Study: Game Effectiveness

Pilot Study: Intervention Effectiveness

Relationship b/w Self-Efficacy and how much students play the game

Self-Efficacy & Grit interact in an interesting way


F(1,105)=3.81 p=0.05


χ2 test, p<.01


F(1,176)=3.46p<0.10


F(1,176)=0.05p>.10


F(1,176)=4.50p<.05


F(1,176)=0.03p>.10

Lessons Learned & Limitations

• Promising directions for adapting to cognitive + non-cognitive factors into games

• Gamification serves a purpose• Countdown timer adds pressure• Speed forces rapid comparisons• “pretty fun for a school game”

• Desire for common game features• Customization, currency

• Assumption: the game is a valid measure of number sense

References• Geary DC, Hoard MK, Nugent L, Bailey DH (2013) “Adolescents’

Functional Numeracy Is Predicted by Their School Entry Number System Knowledge”. PLoS ONE 8(1): e54651.

• Gersten, R., and D. Chard. “Number Sense: Rethinking Arithmetic Instruction for Students with Mathematical Disabilities.” The Journal of Special Education 33.1 (1999): 18-28

Thank you!Questions?

[email protected]

Documents

Ryan Carlson - Developing Number Sense with Play: Lessons in Data Analysis