PRINCIPLES OF COGNITIVE SCIENCE IN EDUCATION
SELF-GENERATION, ERRORS, & FEEDBACK
Janet Metcalfe Columbia University
IES 2006
MS 143 and the Columbia University CASL project
Thanks go to IES for sponsoring this research.
Thanks go especially to Lisa Son, Bridgid Finn and Nate
Kornell
THE GOAL OF THIS PROJECT IS TO USE PRINCIPLES OF COGNITIVE
SCIENCE TO DEVELOP A PROGRAM THAT CAN BE BROADLY USED TO
HELP STUDENTS STUDY. Why do they need it?
(1)self study is vulnerable to metacognitive illusions
(2) certain effective strategies are difficult to implement on one’s own
BACKGROUND
The Bronx project
Strict Lenient0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ComputerSelfNone
Accuracy - Advanced English Vocabulary (English-Speaking Children)
Scoring
Accu
racy
Spanish-speaking children learning English vocabulary
Strict Lenient0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ComputerSelfNone
Accuracy - Basic English Vocabulary (Spanish-Speaking Children)
Scoring
Accuracy
Even Columbia students showed substantial gains, though not as high
as the at-risk children.
Strict Lenient0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
ComputerSelfNone
Accuracy - Basic Spanish Vocabulary (English Speaking CU Students)
Scoring
Accuracy
GOAL: TO ISOLATE COMPONENTS IN THE ORIGINAL TASK THAT ENHANCED OR HARMED LEARNING.
Generation
Errors
Feedback
SELF GENERATION
To our surprise, we, several times, failed to find beneficial effects of self generation.
There was no difference between the control condition and either generation
condition.
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Test 1Test 2
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Test 1Test 2
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
With Columbia students, there was also no generation effect.
So, we tried again, with slight modifications.
Bronx SIXTH GRADE children.
NO GENERATION EFFECT
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Immediate testDelayed test
CONDITION
PR
OP
OR
TIO
N C
OR
REC
T
Columbia students
NO GENERATION EFFECT.
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Immediate testDelayed test
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
WHY NO GENERTION EFFECT ?Our hypotheses were
1. WE GAVE THE CUE FIRST ALONE WITH A 1 SECOND PAUSE WHICH ALLOWED PEOPLE TO GENERATE EVEN IN THE READ CONDITION.
2. THE GENERATE AND READ ITEMS
WERE MIXED WITHIN LIST
If we alter the conditions so people do not generate in the ‘read’ condition, we get a generation effect.
GENERATE AND READ WERE NOW BETWEEN LIST, AND THE CUE AND
TARGET WERE PRESENTED TOGETHER.
0
.05
.1
.15
.2
.25
Ce
ll M
ea
n f
or
Len
ien
t
Immediate Delayed
Retrieval
PassivePres
ActivePres
THIS PRODUCED A LARGE GENERATE EFFECT
SIMILARLY, WITH COLUMBIA STUDENTS
LEARNING GRE WORDS
0
0.1
0.2
0.3
0.4
0.5
0.6
Presentation Retrieval
Condition
Len
ien
t Acc
ura
cy
WE FOUND A LARGE GENERATE EFFECT
PEDAGOGICAL IMPLICATIONS
SELF-GENERATION OF ANSWERS PRODUCES
LARGE LEARNING BENEFITS.
IN A CLASSROOM SITUATION, WHERE ONE CAN’T HAVE EVERY STUDENT GENERATE ALL THE TIME, ONE CAN GET GENERATION BENEFITS BY DOING WHAT WE HAD DONE IN THE FIRST SET OF EXPERIMENTS:
(1)LEAVING A PAUSE BEFORE GIVING THE ANSWER
AND(2) MIXING IN QUESTIONS TO INDIVIDUAL
STUDENTS, WHO MUST THEN GENERATE THE ANSWERS THEMSELVES, FROM TIME TO TIME.
ERRORS
We observed no effect of whether we forced people to generate an answer (which resulted in a huge number of errors) or whether we allowed them to generate only if they felt confident in their answers (which resulted in few errors).
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Test 1Test 2
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
NO EFFFECT OF ERRORSBRONX 6TH GRADE
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Test 1Test 2
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
AGAIN, NO EFFECT OF MAKING ERRORS, COLUMBIA STUDENTS
Bronx SIXTH GRADE AGAINNO EFFECT OF ERRORS
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Immediate testDelayed test
CONDITION
PR
OP
OR
TIO
N C
OR
REC
T
Columbia students: NO EFFECT OF ERRORS
control forced generation free generation0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Immediate testDelayed test
CONDITION
PR
OP
OR
TIO
N C
OR
RE
CT
However, it seems like committing an error should be problematic.
So we ran two more experiments directed at errors specifically.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Errors No errors
Condition
Len
ien
t Acc
ura
cy
BronxChildren were either forced to generate--producing many errors or told to generate, but not to guess, producing few, in a between list design. They were given immediate feedback. Errors made no difference.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Errors No errors
Condition
Len
ien
t Acc
ura
cy
Columbia
The same experiment was run with more difficult materials.
Again, there was no effect.
PEDAGOGICAL IMPLICATIONS
We have no evidence to support the contention that making errors is harmful, as long as corrective feedback is given, even though we have now sought such evidence 6 times.
FEEDBACK
So far, I have shown results only when feedback was given.
But, we also had no feedback conditions.
IN EVERY CASE, FEEDBACK HELPED.
Feedback No Feedback0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Forced (many errors) free (few errors)
Data from Bronx studentsPro
port
ion C
orr
ect
Feedback No Feedback0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
forced (many errors) free (few errors))
Data from Columbia StudentsPro
port
ion
Corr
ect
Feedback No Feedback0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ForcedFree
Data from Columbia, Experiment 2Pro
port
ion
Corr
ect
In all three experiments, the effects of initial feedback also
held up in the second test.
In the final experiment we investigated the issue of how
important it is to give immediate, as compared to
delayed, feedback.
There were five conditions:
Immediate feedback immediate test
No feedback immediate test
Immediate feedback delayed test
Delayed feedback delayed test
No feedback delayed test
The experiment was run over four days. On the fourth day we
did the delayed test.
0
.2
.4
.6
.8
1
Acc
ura
cy
De
l te
st/D
el
fee
db
ack
De
l te
st/I
mm
fe
ed
ba
ck
De
l te
st/N
o f
ee
db
ack
Imm
te
st/I
mm
fe
ed
ba
ck
Imm
te
st/N
o f
ee
db
ack
The design of this experiment allowed us to look at the effect of
immediate versus delayed feedback, holding constant
feedback lag to test (by only using the middle sessions).
delayed immediate no feedback0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Data from Columbia Dragon Masters
Condition
Pro
port
ion C
orr
ect
WE HAVE DONE A REPLICATION STUDY WITH GRADE 6 CHILDREN AT THE SCHOOL AT COLUMBIA UNIVERSITY AND FOUND SIGNIFICANTLY BETTER FINAL TEST PERFORMANCE WHEN FEEDBACK WAS GIVEN AT A DELAY RATHER THAN IMMEDIATELY.
PEDAGOGICAL IMPLICATIONS
We DO need to give feedback, and good feedback that is very specific and that the person fully learns and understands.
BUT, we may not have to give that feedback immediately.
Conclusions(1) Having students self generate the answers rather than passively presenting the answers to them has a large effect.
(2) It makes no difference if the person makes errors in attempting to generate their answers, as long as they are given corrective feedback.
(3) Feedback matters enormously.
(4) It may not matter whether the feedback is given immediately or at a delay--indeed, delayed feedback may be better.