1 LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken Koedinger...
If you can't read please download the document
1 LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director
1 LearnLab: Bridging the Gap Between Learning Science and
Educational Practice Ken Koedinger Human-Computer Interaction &
Psychology, CMU PI & CMU Director of LearnLab
Slide 2
2 Real World Impact of Cognitive Science Algebra Cognitive
Tutor Based on ACT-R theory & cognitive models of student
learning Used in 3000 schools 600,000 students Spin-off: Koedinger,
Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to
school in the big city.
Slide 3
3 Personalized instruction Challenging questions
individualization Progress Authentic problems Feedback within
complex solutions Cognitive Tutors: Interactive Support for
Learning by Doing
Slide 4
4 Success ingredients AI technology Cognitive Task Analysis
Principles of instruction & experimental methods Fast
development & use-driven iteration
Slide 5
Cognitive Task Analysis: What is hard for Algebra students?
Story Problem As a waiter, Ted gets $6 per hour. One night he made
$66 in tips and earned a total of $81.90. How many hours did Ted
work? Word Problem Starting with some number, if I multiply it by 6
and then add 66, I get 81.90. What number did I start with?
Equation x * 6 + 66 = 81.90
Slide 6
6 0 10 20 30 40 50 60 70 80 90 100 Elementary Teachers Middle
School Teachers High School Teachers % Correctly ranking equations
as hardest Nathan & Koedinger (2000). An investigation of
teachers beliefs of students algebra development. Cognition and
Instruction. Expert Blind Spot! Koedinger & Nathan (2004). The
real story behind story problems: Effects of representations on
quantitative reasoning. The Journal of the Learning Sciences. Data
contradicts common beliefs of researchers and teachers
Slide 7
7 Cognitive Tutor Algebra course yields significantly better
learning Course includes text, tutor, teacher professional
development ~11 of 14 full-year controlled studies demonstrate
significantly better student learning Koedinger, Anderson, Hadley,
& Mark (1997). Intelligent tutoring goes to school in the big
city.
Slide 8
8 Success? Yes Done? No! Why not? Student achievement still not
ideal Field study results are imperfect Many design decisions with
no research base Use deployed technology to collect data, make
discoveries, & continually improve
Slide 9
9 PSLC Vision Why? Chasm between science & ed practice
Purpose: Identify the conditions that cause robust student learning
Educational technology as instrument Science-practice collaboration
structure Core Funding: 2004-2014
Slide 10
10 What we know about our own learning What we do not know You
cant design for what you dont know! Do you know what you know?
Slide 11
11 Chemistry Virtual Lab Algebra Cognitive Tutor Ed tech + wide
use = Basic research at scale = Transforming Education R&D
Fundamentally transform Applied research in education Generation of
practice- relevant learning theory + English Grammar Tutor
Educational Games
Slide 12
Ed Tech => Data => Better learning LearnLab Thrusts
LearnLab Course Committees
Slide 13
13 How you can benefit from LearnLab Research General
principles to improve learning Methods Cognitive task analysis, in
vivo studies Technology tools People Masters students &
projects
Slide 14
14 What instructional strategies work best? More assistance vs.
more challenge Basics vs. understanding Education wars in reading,
math, science Koedinger & Aleven (2007). Exploring the
assistance dilemma in experiments with Cognitive Tutors. Ed Psych
Review. Research on many dimensions Massed vs. distributed
(Pashler) Study vs. test (Roediger) Examples vs. problem solving
(Sweller,Renkl) Direct instruction vs. discovery learning (Klahr)
Re-explain vs. ask for explanation (Chi, Renkl) Immediate vs.
delayed (Anderson vs. Bjork) Concrete vs. abstract (Pavio vs.
Kaminski)
Slide 15
15 Knowledge-Learning-Instruction (KLI) Framework: What
conditions cause robust learning LearnLab research thrusts address
KLI elements Cognitive Factors Charles Perfetti, David Klahr
Metacognition & Motivation Vincent Aleven, Tim Nokes-Malach
Social Communication Lauren Resnick, Carolyn Rose Computational
Modeling & Data Mining Geoff Gordon, Ken Koedinger Koedinger et
al. (2012). The Knowledge-Learning- Instruction (KLI) framework:
Bridging the science-practice chasm to enhance robust student
learning. Cognitive Science.
Slide 16
16 Results of ~200 in vivo experiments => Optimal
instruction depends on knowledge goals
Slide 17
17 Cognitive Task Analysis using DataShops learning curve tools
Without decomposition, using just a single Geometry KC, Upshot: Can
automate analysis & produce better student models But with
decomposition, 12 KCs for area concepts, a smoother learning curve.
no smooth learning curve.
Slide 18
18 How you can benefit from LearnLab Research General
principles to improve learning Methods Cognitive task analysis, in
vivo studies Technologies Tutor authoring Language processing
Educational Data Mining People: Masters students &
projects
Slide 19
19 Questions?
Slide 20
20 Question for you What do you need in a learning science
professional?
Slide 21
21
Slide 22
22 Extra slides
Slide 23
23 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive
Tutor Technology Cognitive Model: A system that can solve problems
in the various ways students can If goal is solve a(bx+c) = d Then
rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite
as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c =
d/a Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
Slide 24
24 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive
Tutor Technology Cognitive Model: A system that can solve problems
in the various ways students can If goal is solve a(bx+c) = d Then
rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite
as abx + c = d Model Tracing: Follows student through their
individual approach to a problem -> context-sensitive
instruction Hint message: Distribute a across the parentheses. Bug
message: You need to multiply c by a also. Knowledge Tracing:
Assesses student's knowledge growth -> individualized activity
selection and pacing Known? = 85% chanceKnown? = 45%
Slide 25
25 Cognitive Task Analysis Improves Instruction Studies:
Traditional instruction vs. CTA-based Med school catheter insertion
(Velmahos et al., 2004) Radar system troubleshooting (Schaafstal et
al., 2000) Spreadsheet use (Merrill, 2002) Lee (2004)
meta-analysis: 1.7 effect size!
Slide 26
26 Learning Curves
Slide 27
27 Inspect curves for individual knowledge components (KCs)
Some do not => Opportunity to improve model! Many curves show a
reasonable decline
Slide 28
28 DataShops leaderboard ranks alternative models 100s of
datasets from ed tech in math, science, & language Best model
finds 18 components of knowledge (KCs) that best predict transfer
28
Slide 29
Data from a variety of educational technologies & domains
29 Numberline Game Statistics Online Course English Article Tutor
Algebra Cognitive Tutor
Slide 30
Model discovery across domains 30 11 of 11 improved models
Variety of domains & technologies Koedinger, McLaughlin, &
Stamper (2012). Automated student model improvement. In Proceedings
of Educational Data Mining. [Conference best paper.]
Slide 31
31 Data reveals students achievement & motivations We have
used it to Predict future state test scores as well or better than
the tests themselves Assess dispositions like work ethic Assess
motivation & engagement Assess & improve learning skills
like help seeking
Slide 32
32 LearnLab courses at K12 & College Sites 6 +
cyber-enabled courses: Chemistry, Physics, Algebra, Geometry,
Chinese, English Data collection Students do home/lab work on
tutors, vlab, OLI, Log data, questionnaires, tests DataShop
Researchers Schools Learn Lab Chemistry virtual lab Physics
intelligent tutor REAP vocabulary tutor
Slide 33
33 Lab experiment In Vivo Experiment Design Research Randomzd
Field Trial SettingLabSchoolSchoolSchool Control
conditionYesYesNoYes Focus on principle vs. on solution (Change N
things) Scientific Principle Instr. Solution
Cost/Duration$/Short$$/Medium $$/Long $$$$/Long Bridging
methodology: in vivo experiments
Slide 34
34 Knowledge Components Definition: An acquired unit of
cognitive function or structure that can be inferred from
performance on a set of related tasks Includes: skills, concepts,
schemas, metacognitive strategies, malleable habits of mind,
thinking & learning skills May also include: malleable
motivational beliefs & dispositions Does not include: fixed
cognitive architecture, transient states of cognition or affect
Components of intellectual plasticity Koedinger et al. (2012). The
Knowledge-Learning- Instruction (KLI) framework: Bridging the
science- practice chasm to enhance robust student learning.
Cognitive Science.
Slide 35
35 General knowledge components, sense-making, motivation,
social intelligence Possible domain-general KCs Metacognitive
strategy Novice KC: If Im studying an example, try to remember each
step Desired KC: If Im studying an example, try to explain how each
step follows from the previous Motivational belief Novice: I am no
good at math Desired: I can get better at math by studying &
practicing Social communicative strategy Novice: If an authority
makes a claim, it is true Desired: If considering a claim, look for
evidence for & against it
Slide 36
36 What is Robust Learning? Achieved through: Conceptual
understanding & sense-making skills Refinement of initial
understanding Development of procedural fluency with basic skills
Measured by: Transfer to novel tasks Retention over the long term,
and/or Acceleration of future learning
Slide 37
37 KLI summary Learning occurs in components (KCs) KCs vary in
kind/cmplxty Require different kinds of learning mechanisms Optimal
instructional choices are dependent on KC complexity Intelligence
does not improve generically Koedinger et al. (2012). The
Knowledge-Learning-Instruction (KLI) framework: Bridging the
science-practice chasm to enhance robust student learning.
Cognitive Science.
Slide 38
38 Conclusions Learning & education are complex systems
Lots of work for learning science! Use ed tech for basic research
at scale => Bridge science-practice chasm