Upload
brandon-gibson
View
215
Download
0
Embed Size (px)
Citation preview
1
Studying and achieving robust learning with PSLC resources
Ken KoedingerHCI & Psychology
CMU Director of PSLC
2
This is the 4th Annual PSLC Summer School
• 8th overall– ITS was focus in
2001 to 2004• Goals:
– Learning science & technology concepts
– Hands-on project you present on Fri
Pittsburgh Science of Pittsburgh Science of Learning Center (PSLC)Learning Center (PSLC)170+ researchers from CA to Germany
Ken Koedinger - Carnegie Mellon Co-Director Kurt VanLehn - University of Pittsburgh Co-DirectorCharles Perfetti - Chief Scientist & Future Co-Director
Executive Committee: Vincent Aleven, Maxine Eskenazi (Diversity Director), Julie Fiez, David Klahr (Education Director), Marsha Lovett, Tim Nokes, Lauren Resnick+ representatives of Jr. Faculty, Post-docs & Grad students
Michael Bett – Managing Director
4
Advancing a Science of Academic LearningChallenge: Chasm between learning
science & educational practice • Empirical
– Lots of rigorous principle-testing lab studies– Lots of realistic classroom design research– Too few experiments combine both
• Theoretical– Almost as many theories as there are results!– Need for “rigorous, sustained scientific research in
education” (National Research Council, 2002)
5
Overview
• Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods, Resources, & Theory– In vivo experimentation– LearnLab courses– Robust learning theoretical framework– Enabling technologies
• Summary
Next
6
PSLC is about much more than Intelligent Tutors
But tutors & course evaluations were a key inspiration
Quick review …
7
Past Success: Intelligent Tutors Bring Learning Science to Schools! • Intelligent tutoring
systems– Automated 1:1 tutor– Artificial Intelligence– Cognitive Psychology
• Andes: College Physics Tutor– Replaces homework
• Algebra Cognitive Tutor – Part of complete course
Students: model problems with Students: model problems with diagrams, graphs, equationsdiagrams, graphs, equations
Tutor: feedback, help, Tutor: feedback, help, reflective dialogreflective dialog
b
CognitiveTutorTechnology
Algebra I GeometryEquationSolver
Algebra II
Cognitive Tutors
ArtificialIntelligence
CognitivePsychology
Research base
Math InstructorsMath EducatorsNCTM Standards
Curriculum Content
Cognitive Tutor Approach
9
• Cognitive Model: A system that can solve problems in the various ways students can
Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = dStrategy 2: IF the goal is to solve a(bx+c) = d
THEN rewrite this as bx + c = d/a
Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d
Cognitive Tutor Technology
10
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
11
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
Hint message: “Distribute a across the parentheses.”
Bug message: “You need tomultiply c by a also.”
• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing
Known? = 85% chance Known? = 45%
12
Cognitive Tutor CourseDevelopment Process1. Client & problem identification2. Identify the target task & “interface”3. Perform Cognitive Task Analysis (CTA)4. Create Cognitive Model & Tutor
a. Enhance interface based on CTAb. Create Cognitive Model based on CTAc. Build a curriculum based on CTA
5. Pilot & Parametric Studies6. Classroom Evaluation & Dissemination
b
CognitiveTutorTechnology
Algebra I GeometryEquationSolver
Algebra II
Cognitive Tutors
ArtificialIntelligence
CognitivePsychology
Research base
Math InstructorsMath EducatorsNCTM Standards
Curriculum Content
Cognitive Tutor Approach
14
Difficulty Factors Assessment:Discovering What is Hard for Students to Learn
Which problem type is most difficult for Algebra students?
Story Problem
As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?
Word Problem
Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?
Equation
x * 6 + 66 = 81.90
15
Algebra Student Results:Story Problems are Easier!
70%61%
42%
0%
20%
40%
60%
80%
Story Word Equation
Problem Representation
Percent Correct
Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.
Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science.
16
Expert Blind Spot:Expertise can impair judgment of student difficulties
0
10
20
30
40
50
60
70
80
90
100
ElementaryTeachers
MiddleSchool
Teachers
High SchoolTeachers
% makingcorrectranking
(equations hardest)
17
“The Student Is Not Like Me”
• To avoid your expert blindspot, remember the mantra:
“The Student Is Not Like Me”
• Perform Cognitive Task Analysis to find out what students are like
18
Cognitive Tutor CourseDevelopment Process1. Client & problem identification2. Identify the target task & “interface”3. Perform Cognitive Task Analysis (CTA)4. Create Cognitive Model & Tutor
a. Enhance interface based on CTAb. Create Cognitive Model based on CTAc. Build a curriculum based on CTA
5. Pilot & Parametric Studies6. Classroom Evaluation & Dissemination
19
Tutors make a significant difference in improving student learning!
• Andes: College Physics Tutor– Field studies: Significant
improvements in student learning
• Algebra Cognitive Tutor – 10+ full year field
studies: improvements on problem solving, concepts, basic skills
– Regularly used in 1000s of schools by 100,000s of students!!
0
10
20
30
40
50
60
Iowa SAT subset ProblemSolving
Represent-ations
Traditional Algebra Course
Cognitive Tutor Algebra
20
Prior achievement:Intelligent Tutoring Systems bring learning science to schools
A key PSLC inspiration:Educational technology as research platform to generate new learning science
21
Logic of Pittsburgh Science of Learning Center (PSLC)• Support experimental studies that
– Test fundamental principles, not whole courses
– Are internally & externally valid
• Create a theory of “robust learning”• Leverage technology &
computational modeling
22
Overview
• Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods, Resources, & Theory– In vivo experimentation– LearnLab courses– Robust learning theoretical framework– Enabling technologies
• Summary
Next
23
PSLC statement of purpose
• To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation.
Outline of Overview
1. Robust learning
2. Theory
3. Method
4. LearnLab
Next
24
What is Robust Learning?
• Robust Learning is learning that – transfers to novel tasks– retained over the long term, and/or – accelerates future learning
• Robust learning requires that students develop both– conceptual understanding & sense-making skills– procedural fluency with basic foundational skills
25
PSLC statement of purpose
• To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation.
Outline of Overview
1. Robust learning
2. Theory
3. Method
4. LearnLab
Next
26
In Vivo Experiments Principle-testing laboratory quality in real classrooms
27
In Vivo Experimentation Methodology
Methodology features:• What is tested?
– Instructional solution vs. causal principle
• Where & who?– Lab vs. classroom
• How?– Treatment only vs.
Treatment + control
Lab experiments
Design research
& field trials
Instructional solution
Cla
ssro
om
La
b
Causal principle
What is tested?
Wh
e re ?
• Generalizing conclusions:– Ecological validity: What instructional
activities work in real classrooms?– Internal validity: What causal
mechanisms explain & predict?
28
Methodology features:• What is tested?
– Instructional solution vs. causal principle
• Where & who?– Lab vs. classroom
• How?– Treatment only vs.
Treatment + control
Lab experiments
Design research
& field trials
In Vivo Experimentation Methodology
In Vivo learning
experiments
Instructional solution
Causal principle
What is tested?
Wh
e re ?
• Generalizing conclusions:– Ecological validity: What instructional
activities work in real classrooms?– Internal validity: What causal
mechanisms explain & predict?
Cla
ssro
om
La
b
29
PSLC statement of purpose
• To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation.
Outline of Overview
1. Robust learning
2. Theory
3. Method
4. LearnLab
Next
30
LearnLabA Facility for Principle-Testing
Experiments in Classrooms
31
LearnLab courses at K12 & College Sites
• 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English
• Data collection– Students do home/lab work
on tutors, vlab, OLI, …– Log data, questionnaires,
tests DataShop
Researchers Schools
Learn Lab
Chemistry virtual labChemistry virtual lab
Physics intelligent tutorPhysics intelligent tutor
REAP vocabolary tutorREAP vocabolary tutor
32
PSLC resources enable theoretical development …
33
PSLC statement of purpose
• To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation.
Outline of Overview
1. Robust learning
2. Theory
3. Method
4. LearnLab
Next
34
Theoretical Framework Levels• Macro level
– What instructional principles explain how changes in the instructional environment lead to changes in robust learning?
• Micro level
35
Explanation at the macro-level
Novice knowledge state
Learning Processes
Instructional Method IT vs. IC
Pre-test Accelerated future learning
Transfer Long-term retention
Robust Learning measures:
Expert knowledge state
Normal Post-test
Why is instruction IT better than IC?
36
Hausmann & VanLehn 2007 study: Macro level description• Research question:
Does providing explanations or eliciting “self-explanations” from students better enhance robust learning?
• General instruction: Students alternate between– Watching videos of worked examples of physics
problems– Solving new problems in the Andes intelligent tutor
• Treatment variables: – Videos include justifications for steps or do not– Students are prompted to “self-explain” or paraphrase
37
Self-explanations => greater robust learning
0.670.670.90 0.980
0.2
0.4
0.6
0.8
1
1.2
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
• Justifications: no effect!
• Immediate test on electricity problems:
• Transfer to new electricity homework problems
0.450.370.69 1.040
0.2
0.4
0.6
0.8
1
1.2
1.4
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
• Instruction on electricity unit => accelerated future learning of magnetism!
0.911.17 0.75 0.680
0.2
0.4
0.6
0.8
1
1.2
1.4
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
38
Key features of H&V study
• In vivo experiment– Ran live in 4 physics sections at US Naval
Academy – Principle-focused: 2x2 single treatment
variations– Tight control manipulated through
technology
• Use of Andes tutor => repeated embedded assessment without
disrupting course
• Data in DataShop (more later)
39
Theory Integration Strategy
• PSLC has supported some 130+ studies
• How do they fit together?
40
Robust Learning
Sense-Making
Foundational Skills
Macro-level of robust learning framework: Taxonomy of outcomes, processes, & treatments
Outcomes:
Instructional Treatments:
Collaboration, tell vs. ask, question insertion
Visual-verbal coordination, example-rule coordination
Explicit instruction,
learner control…
Schedules, …
Construction, elaboration,
discriminationCo-Training Refinement
of FeaturesStreng-thening
Learning Processes:
41
Robust Learning
Sense-Making
Foundational Skills
3 Original Research clusters
Outcomes:
Instructional Treatments:
Collaboration, tell vs. ask, question insertion
Visual-verbal coordination, example-rule coordination
Explicit instruction,
learner control…
Schedules, …
Construction, elaboration,
discriminationCo-Training Refinement
of FeaturesStreng-thening
Learning Processes:
Interactive Interactive CommunicationCommunication
Coordinative Coordinative LearningLearning
Fluency and Fluency and RefinementRefinement
42
Robust Learning
Sense-Making
Foundational Skills
Hausmann & VanLehn in Interactive Communication Cluster
Outcomes:
Instructional Treatments:
Collaboration, tell vs. ask, question insertion
Explicit instruction,
learner control…
Schedules, …
Visual-verbal coordination, example-rule coordination
Construction, elaboration,
discriminationCo-Training Refinement
of FeaturesStreng-thening
Learning Processes:
Interactive Interactive CommunicationCommunication
Coordinative Coordinative LearningLearning
Fluency and Fluency and RefinementRefinement
43
Interactive Communication Research Question• What properties of interactive
communication promote robust learning?
– 20+ projects/studies have addressed versions of this question
44
Use of wiki toward macro-level theory developmentExperimenters & instructors:• Create study summary pages on wiki• Work out shared meanings of terms
– Like, what does “self-explanation” mean?– Create & edit glossary pages in the wiki
• Seek out commonalities with others – Tie their hypotheses to general principles– Create shared instructional principle pages
45
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
PSLC wiki supports theory integration
Links to H&V study page:Interactive Communication cluster page:
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
46
PSLC wiki supports theory integration
182 concepts in glossarySelf-explanation glossary entry
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
47
PSLC wiki supports theory integration
Points back to Hausmann’s study page
Instructional Principle pages unify across studies
48
Theoretical Framework Levels• Macro level
– What instructional principles explain how changes in the instructional environment lead to changes in robust learning?
• Micro level– Can learning be explained in terms of what
knowledge components are acquired at individual learning events?
49
Key Micro-level Concepts
• Knowledge component (unobservable)– A mental structure or process that a learner uses to
accomplish steps in a task or a problem
• Learning event (unobservable)– Points in time where knowledge components are
learned or used
• Instructional event (observable)– Element of learning environment designed to evoke
a particular learning event– P(Learning_event | Instructional_event) < 1.0
• Learner factors like metacognition & motivation affect this conditional probability
50
Kinds of Knowledge Components (KCs)Mental representations of:• Domain knowledge
– Facts, concepts, principles, rules, procedures, strategies
• Prerequisite knowledge– Feature encoding knowledge
• Integrative knowledge– Schemas or procedures that
connect other KCs• Metacognitive knowledge
– About knowledge or controlling use of knowledge
• Beliefs & interests– What one likes, believes
• Cross-cutting distinctions– Correct vs. incorrect – Verbal (explicit) vs. non-verbal
(implicit)– Probabilistic vs. discrete
NOT KCs:• Any external representation of
knowledge– Textbook descriptions
• Generic cognitive structures– Working memory
• Continuous parameters on knowledge representations
– Strength, level of engagement, implicit value of a goal, affect
51
Macro-level analysis treats knowledge states & learning processes as black boxes
Novice knowledge state
Learning Processes
Instructional Method IT vs. IC
Pre-test Accelerated future learning
Transfer Long-term retention
Robust Learning measures:
Expert knowledge state
Normal Post-test
52
Macro-level analysis treats knowledge states & learning processes as black boxes
Novice knowledge state
Learning Processes
Instructional Method IT vs. IC
Pre-test Accelerated future learning
Transfer Long-term retention
Robust Learning measures:
Expert knowledge state
Normal Post-test
53
Unpacking learning: 3 kinds of events
Novice knowledge state
Pre-testRobust Learning measures
Expert (desired) knowledge state
Normal Post-test
Learning Event
Instructional Event
Assessment event
Learning Event
Instructional Event
Assessment event
Learning Event
Instructional Event
54
PSLC’s DataShop
• Vast open data repository, analysis tools, visualizations
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
• Generates:– learning curves– statistical model
fit (blue line)
• Based on micro level analysis:– learning event
opportunities– Averaged across
knowledge components
55
Greatest impact of technology on 21st century education?
• Benefits to student learning from use of educational technology?
• Perhaps• But my bet is:
– Scientific advances coming from data mining of the vast explosion of learning data that will be coming from educational technologies
56
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Problem1 Problem2 Problem3
(hints+errors)/steps
Instructionalexplanation
Self-explanation
Example1 Example2 Example3
Back to H&V study: Micro-analysisLearning curve for main KCSelf-explanation effect tapers but not to zero
57
Overview
• Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods, Resources, & Theory– In vivo experimentation– LearnLab courses– Robust learning theoretical framework– Enabling technologies
• Summary Next
58
PSLC Enabling Technologies
• Tools for developing instruction & experiments– CTAT (cognitive tutoring systems)
• SimStudent (generalizing an example-tracing tutor)
– OLI (learning management)– TuTalk (natural language dialogue)– REAP (authentic texts)
• Tools for data analysis – DataShop– TagHelper
59
Summary• Learning can be greatly improved!
– Address robust learning– Increase certainty– Leverage technology
• PSLC provides tools & processes to help learning researchersachieve these goals
RobustLearning
Principles
Existing SoloTheories
In VivoStudies
EnablingTechnology
LearnLabCourses
60
END