Tracing behaviors associated with motivational states and learning outcomes

when students learn with the Cognitive Tutor

Team: Matthew Bernacki & Pranav Garg

Mentors: Erik Zawadzki & Ryan Baker

20

11

Sum

mer

Sch

ool

Overview

• We investigated relationships between motivation, learning behaviors and learning outcomes amongst high school students learning geometry using the Cognitive Tutor.

• We identified a series of 3 sequential behaviors (triplets) and plotted their frequency across the logs of 38 learners in one geometry unit.

• We conducted a factor analysis to reduce 147 triplets into 28 factors and examined their correlation with self-reports of affective state, self-efficacy for the unit and their achievement goals for mathematics.

METHOD: In the Classroom

Participants• 38 high school geometry students completing Unit 13 in the

Cognitive Tutor, which was a standard component of the their rural high school’s geometry curriculum.

Instruments• Cognitive Tutor for Geometry

– Unit 13: Circumference and Area of Circles

• Achievement Goal Questionnaire-Revised– Elliot & Murayama, 2008 (9 items, 3 per Mastery Approach, Performance

Approach, Performance Avoidance subscale)

• Academic Self Efficacy Survey– Midgely, et al., 2000; Patterns of Adaptive Learning Survey

• Affect (single items constructed for this project)– Boredom, Confusion, Frustration Engaged Concentration,

Positive Experience

METHOD: Data Mining Procedure

1. Exported transaction level log file from Cognitive Tutor2. Selected only those students who completed the Unit of interest;

cleaned data to remove any students who were missing self-report data or a complete log file

3. Calculated the duration (seconds) to complete each learner action in the OUTCOME column

– OK – answered problem step correctly– BUG – incorrectly answered the problem step (common error)– ERROR – incorrectly answered the problem step– HINT [1,2,3]– requested a hint – SWITCH – switched their window to consult a worked example

4. Recoded Duration by Quartile (1, middle 2&3, 4)1. Q1 = Short durations, typically 1-2 seconds; coded as “…_1”2. Q2&3 = Medium Durations, typically 2-10 seconds; coded as “…_2”3. Q4 = Longest durations, typically upward of 10 seconds; coded as “…_3”

5. Concatenated Outcome with Q(uartile version of) Duration.

METHOD: Data Mining Cont’d.

6. Ran a script in Python to move a sliding window over the OutcomeQDuration column and populated a column with a triplet: [FIRST TRANSACTION_DURATION_SECOND_D_THIRD_D].

7. Calculated the total number of unique triplets (n = 7,885) and, with a Pivot Table, determined the frequency each occurred per student.

8. Eliminated those that occurred less than 5 times and those that occurred in less than 2 students (n = 147)

9. Imported into SPSS, merged with a file of their self-reported motivational states and official record of learning outcomes

10. Ran a Principle Components Factor Analysis (unrotated) to determine a factor structure.

11. Correlated Factor Scores with motivation and performance data

RESULTSTHOSE WHO …TEND TO CONDUCT

BEHAVIORS THAT LOAD ON FACTOR

…EXPERIENCE …

Boredom 15, 28

Confusion 27

Frustration 12, 13, 27

Self-Efficacy 6,22

…PURSUE…

Mastery or Performance Approach Goals

27

Performance Avoidance Goals 17

… PERFORM WELL ACCORDING TO…

4th Quarter Grades 3

  Academic Self Efficacy

Boredom Confusion Frustration Positive Experience

Engaged Concentration

Mastery Approach Goals

Performance Approach Goals

Performance Avoidance Goals

3rd Qtr Grade

4th Qtr Grade

FACTOR 1 -.004 .105 -.025 .083 .063 .069 .020 .079 -.114 .099 .032

2 .022 -.045 -.124 -.050 -.238 -.244 -.096 -.258 -.193 .117 .086

3 .220 .090 -.005 .006 .245 .222 .164 .110 -.135 .171 .321*

4 -.088 .125 .077 .082 .078 .127 -.065 .066 .152 -.023 -.166

5 .002 .117 .048 .073 .065 .132 -.108 -.090 -.147 .022 -.001

6 .378* -.086 -.152 -.097 .284 .279 -.126 -.245 -.158 .254 .315

7 .015 -.079 -.277 -.103 .006 -.063 -.108 .017 -.065 .106 .051

8 -.067 .196 .133 .177 -.148 -.165 -.210 -.174 -.156 .116 -.028

9 .054 .095 -.021 .026 -.148 -.116 -.133 .022 .111 .079 .149

10 -.256 -.317 -.007 -.086 .036 .032 .060 .213 .100 -.101 -.148

11 -.133 -.051 .224 .120 .032 .069 .009 -.037 .099 .068 .237

12 -.005 -.136 -.194 -.452** -.179 -.178 -.108 .202 .030 .212 .118

13 .113 -.203 .369* .398* .262 .159 -.003 -.187 -.165 -.249 -.197

14 -.057 -.199 -.316 -.131 .090 .002 -.267 -.056 -.011 -.083 -.099

15 -.146 .354* .119 .194 -.091 -.051 .219 -.053 -.094 -.168 -.073

16 .138 -.039 .189 .228 .086 .109 -.098 -.108 .158 -.030 -.133

17 .147 .076 .189 .111 .261 .204 .044 .045 .373* .194 .316

18 .084 -.034 -.035 -.057 .016 -.027 -.244 -.004 -.101 -.109 -.246

19 .145 -.056 .165 .095 .102 .070 -.092 -.120 .001 .152 .005

20 .098 -.052 -.153 -.178 -.275 -.217 .125 .064 .127 .162 .049

21 -.243 -.111 .107 .160 .002 .034 .141 -.041 .159 .213 .217

22 -.329* -.072 -.087 -.063 -.061 -.063 .161 -.240 -.160 -.114 -.109

23 .043 .178 -.212 -.191 -.216 -.054 -.134 .030 .100 .042 .059

24 -.221 .009 -.002 .150 -.261 -.264 .099 .029 -.111 .212 -.070

25 -.050 -.013 .040 -.021 -.104 -.195 -.101 -.068 .151 .068 .008

26 -.131 -.226 .126 .124 .030 .119 -.298 .281 .139 .008 .05627 .063 -.195 -.417** -.389* .020 -.042 -.599** -.519** -.092 .173 .179

28 .015 .456** .016 .048 -.232 -.193 .042 -.079 .115 .006 -.247

THE FACTORSfactor

Behavior Triplet with highest factor loading

2nd MAX 3rd MAX

1['ERROR_3_ERROR_3_OK_3{.858} , ', 'ERROR_2_OK_3_BUG_2{.858} , ', 'OK_2_BUG_1_OK_2{.858} , ']

['OK_3_ERROR_2_ERROR_2{.845} , ']

['ERROR_2_ERROR_1_ERROR_1{.835} , ', 'ERROR_1_ERROR_1_ERROR_1{.835} , ', 'ERROR_2_ERROR_2_ERROR_1{.835} , ']

2 ['OK_2_OK_3_OK_3{.825} , '] ['OK_3_OK_3_OK_2{.758} , '] ['OK_1_OK_3_OK_2{.746} , ']

3 ['OK_3_ERROR_3_OK_3{.623} , '] ['hint_2_HINT2_1_hint_3{.542} , ', 'OK_3_BUG_2_OK_2{.542} , ']

['OK_3_hint_3_HINT2_2{.520} , ']

4 ['OK_3_OK_3_BUG_3{.738} , '] ['OK_1_OK_2_OK_2{.616} , '] ['OK_1_OK_1_OK_2{.584} , ']

5 ['ERROR_2_HINT_2{.647} , ', 'ERROR_3_OK_3_BUG_3{.647} , ']

['ERROR_3_OK_2_OK_2{.600} , '] ['OK_2_OK_3_BUG_2{.591} , ']

6 ['OK_3_ERROR_3_ERROR_3{.540} , ']

['OK_3_OK_1_OK_3{.470} , '] ['OK_3_OK_3_ERROR_3{.467} , ']

7 ['OK_1_OK_1_BUG_3{.599} , '] ['HINT3_2_ERROR_3_ERROR_2{.582} , ']

['OK_2_OK_3_BUG_1{.551} , ']

8 ['BUG_2_OK_2_OK_2{.612} , '] ['OK_2_OK_2_BUG_2{.480} , '] ['OK_1_OK_2_BUG_1{.453} , ']

9 ['OK_1_OK_1_BUG_1{.603} , '] ['OK_2_BUG_2_OK_2{.500} , '] ['OK_3_OK_3_OK_1{.483} , ']

10 ['BUG_3_OK_3_OK_1{.465} , '] ['OK_2_BUG_1_OK_3{.457} , '] ['HINT3_3_OK_3_OK_2{.448} , ']

11 ['HINT_2 _HINT_1_HINT2_1{.560} , ']

['OK_3_ERROR_3_ERROR_3{.437} , ']

['ERROR_2_OK_3_ERROR_2{.433} , ']

12 ['BUG_1_OK_3_OK_2{.490} , '] ['BUG_3_OK_2_OK_3{.477} , '] ['ERROR_3_OK_3_OK_3{.382} , ']

13 ['OK_3_BUG_1_OK_2{.518} , '] ['HINT3_2_ERROR_3_ERROR_2{.476} , ']

['OK_1_OK_3_BUG_1{.430} , ']

14 ['OK_2_OK_3_ERROR_3{.437} , '] ['OK_3_ERROR_2_OK_2{.387} , '] ['BUG_3_OK_2_OK_2{.349} , ']

THE FACTORSfactor

Behavior Triplet with highest factor loading

2nd MAX 3rd MAX

15 ['HINT2_1_hint_3_HINT2_2{.575} , ']

['hint_3_HINT2_1_HINT3_3{.451} ']

['BUG_3_OK_3_OK_3{.401} , ']

16 ['ERROR_3_OK_3_OK_2{.463} , '] ['OK_3_ERROR_2_OK_2{.424} , '] ['OK_3_ERROR_2_ERROR_3{.372} , ']

17 ['OK_2_OK_3_hint_3{.501} , '] ['OK_2_ERROR_2_OK_3{.459} , '] ['OK_2_OK_2_BUG_2{.374} , ']

18 ['BUG_3_OK_3_OK_1{.396} , '] ['OK_1_ERROR_3_OK_3{.356} , '] ['HINT3_2_ERROR_3_ERROR_2{.327} , ']

19 ['OK_3_OK_1_OK_1{.377} , '] ['BUG_3_OK_2_OK_3{.350} , '] ['OK_2_OK_3_OK_2{.337} , ']

20 ['BUG_1_OK_3_OK_2{.373} , '] ['ERROR_3_OK_2_OK_3{.372} , '] ['OK_3_OK_2_OK_1{.319} , ']

21 ['OK_3_BUG_1_OK_3{.382} , '] ['BUG_3_OK_2_OK_3{.312} , '] ['BUG_1_OK_3_OK_2{-.395} , ']

22 ['ERROR_2_ERROR_1_OK_3{.337} , ']

['OK_3_BUG_1_OK_2{.330} , '] ['OK_1_OK_1_ERROR_2{.328} , ']

23 ['OK_1_BUG_3_OK_2{.488} , '] ['ERROR_3_OK_3_BUG_2{.430} , ']

['OK_2_OK_3_ERROR_3{.331} , ']

24 ['OK_2_OK_1_OK_3{.420} , '] ['OK_2_OK_3_BUG_3{.306} , '] ['HINT2_1_HINT3_3_OK_3{-.343} , ']

25 NULL* NULL NULL

26 NULL* NULL NULL

27 ['OK_1_OK_3_OK_1{.306} , '] NULL NULL

28 NULL* NULL NULL

THEORETICAL CONCLUSIONS

• Some behaviors associated with a factor can be interpretted somewhat easily. Factor 12:

• BUG_1_OK_3_OK_2{.490}• BUG_3_OK_2_OK_3{.477} • ERROR_3_OK_3_OK_3{.382}

– Student made errors, often after some perserveration, then correctly answered items after a medium to long period.

– Factor is associated with low frustration.

THEORETICAL CONCLUSIONS

• However, scores on one factor (#27) were significantly associated with self-reports of confusion and frustration and negatively associated with mastery and performance approach goals.– Only one triplet [OK_1_OK_3_OK_1] loaded

higher than .30 on the factor– Low factor loading and meaningfulness of a

short period prior to a correct, followed by a long and a short actually run counter to some conclusions based on self-reports.

METHODOLOGICAL CONCLUSIONS

• Triplets composed of behaviors and their durations can be meaningful measures of behavior

• They can also be insufficiently descriptive of a students’ behavior re: specificity – number of behaviors captured– Precision of duration when collapsed to

quartile– Meaningfulness of cuts between quartiles

NEXT STEPS

• Test factor structure across additional units– If not, may make sense to abandon factors and examine

relations one behavioral trace at a time

• Generate 4-lets and 5-lets to see if these behaviors provide more intuitive glimpses of student behaviors

• Once a set of behaviors has been found that associate with motivation – develop a flag for a behavior… and an intervention?

• Test structural models with paths from motivational state to behavior to learning outcomes