Uncovering the Problem-Solving Process:

Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer

I3CLEPS Workshop/Mini-conference,

August 29, 2005

Cued Retrospective Reporting, Eye Tracking, and Expertise Differences

Overview

Experiment:

- Theory

- Design

Comparison of 3 verbal methods

The 3 methods & expertise differences

Uncovering expertise-related performance differences through eye movement data

Present limitations and future research

Discussion

Theory

Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform:

- Psychological theory

- Expert systems

- User-system interaction,

But also

- Instructional design e.g., design of process-oriented worked examples

Theory

From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000):+ of concurrent reporting (“think aloud”):

more information on actions taken+ of retrospective reporting:

more information on rationale for actions taken and strategies that control the process

Needed: A method that combines + & + :Cued retrospective reporting based on a record of eye

movements & mouse/keyboard operations?

Design

Within-subjects, 26 participants, electrical circuits troubleshooting tasks:

Seq. Condition + Tasks1 CR 1+2 CRE 3+4 RR 5+6 CRR 7+82 CRE 3+4 CRR 7+8 CR 1+2 RR 5+63 RR 5+6 CR 1+2 CRR 7+8 CRE 3+44 CRR 7+8 RR 5+6 CRE 3+4 CR 1+2

CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.

Comparison of 3 Methods: Hypotheses

1. Concurrent reporting (CR): more ‘action’ info than RR

2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR

3. Cued retrospective reporting (CRR):-> more ‘action’ than RR-> more ‘why’, ‘how’, & ‘metacognitive’ than CR

Comparison of 3 Methods: Analyses

Segmentation based on speech sentences / utterances (preceded & followed by a pause)

Coding scheme task-oriented main categories:‘action’‘why’‘how’‘metacognitive’

20% of protocols scored by 2 raters: kappa = .79 good; proceeded with 1 rater

Analyses on nr. of codes on main categories, obtained by summing codes on subcategories

Comparison of 3 Methods: Results

Friedman Tests with Conover (1999) comparisons

CR vs RR:as hypothesized: ‘action’ CR >RRhowever: ‘why’ and ‘how’ CR > RR, and‘metacognitive’ CR = RR

CRR vs RR:as hypothesized: ‘action’ CRR >RR‘why’: CRR = RR‘how’ and ‘metacognitive’: CRR > RR

Expertise Differences: Explorative

5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency:

“highest”: higher performance, lower mental effort, lower time-on-task

“lowest”: lower performance, higher mental effort, higher time-on-task

- Differences in elicited information?

- Differences in preferences/experiences?(open-ended debriefing questions)

Expertise Differences: Elicited Information

Differences in elicited information?(Mann-Whitney U Tests)

CR:

‘how’ and ‘metacognitive’ info: “lowest” > “highest”

RR:

‘why’ info: “highest”> “lowest”

‘how’ info: “lowest” > “highest”

CRR:

‘action’ and ‘metacognitive’ info: “lowest” > “highest”

Expertise Differences: Experience

Differences in preferences/experiences?“lowest”:

experience: CR (4/5)preference: CRR > CR & RR (4/5)

“highest”:no differential experiences/preferences

Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”:

- Time-on-task- Cue

Studying Expertise-Related Performance

Differences: Eye Movement Data 1

Eye fixation data provide insight in the allocation of attention, and hence differ with expertise

Research use: provide information about the problem-solving process at a finer grained level than verbal protocols?

(Ultimate) educational use: guiding novices’ attention?

1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology

Eye Movement Data: Participants & Procedure

Same 5 “lowest” and 5 “highest” expertise participants

Data collected in first 3 phases of the process:

1. Problem orientation (until pushing switch to observe circuit behavior)

2. Problem formulation and action decision

3. Action evaluation and next action decision

% time spent on phase, mean fixation duration (MFD), and in 1st phase fix. related to faults

Only 3 Volt

Short-circuit

Task

Eye Movement Data: Results

Phase 1: problem orientation(Mann-Whitney U Tests, 2-tailed, α = .10)

% of time: “highest” > “lowest”

MFD: “lowest” > “highest”

% fixations on battery: “highest” > “lowest”

Gaze switches short-circuit: “highest” > “lowest”(NB: only trend)

Eye Movement Data: Results

Phase 2: problem formulation & action decision(Mann-Whitney U Tests)

% of time: “highest” = “lowest”

MFD: “highest” = “lowest”

MFD First ½: “highest” > “lowest”

MFD Second ½: “highest” = “lowest”

Eye Movement Data: Results

Phase 3: action evaluation & next action decision(Mann-Whitney U Tests)

% of time: “highest” > “lowest”

MFD: “highest” = “lowest”

MFD First ½: “highest” = “lowest”

MFD Second ½: “highest” = “lowest”

Eye Movement Data: Results

MFD over phases (Friedman + Nemenyi post-hoc):

n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1

Limitations

- CRR and fabrication?- Cue: combination of eye movements AND

mouse/keyboard operations- Only quantitative analyses of protocols- Eye movement data: distinction of phases

- Performance efficiency measure:very relative distinction (lowest and highest within this group of participants)

- Small nr of participants in analyses related to expertise differences

Future Research

- Qualitative differences between CRR and RR?

- Cue: different effects with only eye movements OR mouse/keyboard operations?

- Cue: technical optimization?

- (RR/)CRR: effects of other prompts?

- Further study of performance efficiency measure to distinguish expertise levels

- Replications with larger N

Thank you for your attention!

tamara.vangog@ou.nl