View
475
Download
0
Category
Tags:
Preview:
DESCRIPTION
The recent explosion of online educational resources has the potential to reorganize how we learn – from K-12 and university to the workplace and the informal learning we do every day. It also raises new questions and opportunities for research that crosses the many disciplines relevant to designing computer programs that help people learn. For example, HCI and cognitive science can provide complementary perspectives in investigating how to design the content and instructional features of an online course, such that a person processes and stores that information in a way that successfully guides their future behavior. Online educational environments provide new optimism in tackling challenges like these because they can be instrumented to collect an unprecedented scale and diversity of data, and allow iterative sequences of experiments to be embedded in authentic educational contexts with real students. This talk presents one approach to this kind of research, using experimental comparisons to test the effects of modifying online mathematics exercises to include motivational messages and question prompts for people to explain, the design of which is guided by the psychological literature on motivation and learning. A combination of laboratory experiments and experiments embedded in real-world online education platforms (like www.KhanAcademy.org) reveal that prompting people to explain “why?” facts are true drives them beyond memorization to uncover underlying principles and patterns, and that teaching such self-questioning strategies may accelerate student learning. Motivational messages appear to have limited benefits if they are simply encouraging or aimed at raising confidence, but do increase how much effort students invest if the messages emphasize that aptitude is malleable and can be improved through persistence. Several planned experiments are presented which also use this paradigm of adding minimal but effective textual changes to online exercises to achieve practical impact and explore basic cognitive science questions about learning.
Citation preview
1
Using Experiments and Cognitive Science Research to Improve the Design of
Online Resources for Learning
Joseph Jay Williams
josephjaywilliams@stanford.edu www.josephjaywilliams.com/research-overview
Online Education & Learning Online
• New research area?• Convergence of
computational & behavioral scienceNIPS “Data-Driven Education”New ACM conference “Learning at Scale”CHI
Novel Research Opportunity: Real-World + Laboratory
Integrate Research & Practice
• Randomized assignment• Experimental Control• Rich data
• Generalizable theories• “in vivo” experiments• Diverse populations
• Practical improvements• Disseminate research• Generate Funding
• Real-world environment
• Authentic activities• Practical Challenges
5
Overview
• Explanation & Learning• Teaching Learning Strategies• Motivational Messages• Experimental Paradigm• Experiment-focused Design
6
Why does explaining “why?” help learning?
• General boost to Learning Engagement vs.
• The Subsumptive Constraints Account: Interpret target of why-explanation in terms of a broader generalization (Williams & Lombrozo, 2010)
• 2 x 3 = 6• Why?
Explanation and Learning: Lab Online
• Discovery & transfer (Williams & Lombrozo, 2010,
Cognitive Science)
• Use of prior knowledge (Williams & Lombrozo, 2013,
Cog. Psych.)
• Erroneously overgeneralize at expense of exceptions (Williams et al, 2013, JEP:
General)
• Promotes belief revision – given sufficient anomalies (Williams, Walker, Maldonado &
Lombrozo, 2012; 2013, Cog Sci Conference; in prep)
• Prompts in online (math) exercises (Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing)
GLORP DRENT
8
Learning Task & Experimental Paradigm
• Online (Math) Exercise
1. Number of Problems Completed
2. Percent Correct
9
Logic of my Previous Research
Explain why this is correct.
Elaborate on what you are thinking now.
• Post-Study test questions• Transfer/Generalization
questions• Questions about key principle• Memory for details
10
Future: Generate, Receive, Compare
Geza KovacsSimultaneously Learning AND Crowdsourcing Improvement of Learning Resources
Williams, Thille, Siemens, Trumbore, Stigler. How online resources can facilitate interdisciplinary collaboration. Invited talk to be presented at SIG on Computer and Internet Applications in Education, AERA 2014.
E.g. Explain why this is correct.
Generate
E.g. Another student said: This is correct because…
Receive
E.g. What are the similarities and differences?
Compare
Teaching Learning Strategies
• Spend time teaching specific content, or general strategies?
• Online: Collect data that would be extremely difficult to get in the real world
• Online: Repeatedly reinforce habits & educational behaviors
• Teach “What? Why? How?” self-questioning/explanation strategies (Palinscar & Brown,
1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science)
• Understanding vs. Problem-solving • vs. Interpreting vs. Practice-as-Usual
11
Experimentally manipulate additional prompts
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
+ Prompts embedded into hints
Self-questioning strategy: What? Why? How?
Embedded Prompts between Hint/Solution Steps
Prompts
Prompt type
Prompted-understanding
Prompted-problem-solving
Prompted-interpretation
What? What does this step
mean to you?
What are you doing or thinking right
now?
What is this step saying? Restate it in
your own words.
Why? Why is it helpful to
take this step?
Why is what you are currently doing
helpful? Why is it useful for achieving
your goal?
How? How do you know this step is right?
How well is your current approach to
this problem working?
16
Explanation studyPractice-as-
usual Control Self-Regulating Thinking
Reflecting on Meaning
Problems Attempted
Problems Correct
Practice Tasks Completed
Mastery Tasks Completed
Problem Accuracy
17
Explanation studyPractice-as-usual No Textbox Textbox
Problems Attempted
Problems Correct
Practice Tasks Completed
Mastery Tasks Completed
Problem Accuracy
Learning Behavior Support
• Clickable link to Drop-Down text with suite of strategies:Are you stuck?Click here for some tips.
• Provide previously examined prompts. • Use mouseover and drop-down text to
reveal information “as requested”, rich traversal of options, guided by student
Natural Link to Learning Strategy Training
• If you want to learn more about strategies to keep motivated and learn well, go to tiny.cc/learningassistant or XX or YY
20
3. Add motivational messages
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Message
21
3. Embedded in vivo Experiment
• Growth Mindset Message
• "Remember, the more you practice the smarter you become.”,
• "Mistakes help you learn. Think hard to learn from them.”
• Practice-as-usual
• Benefit of Growth Mindset Message?
Jascha Sohl-Dickstein
22
Results: More motivated?
• Growth Mindset Message > Practice-as-Usual
• extra problems attempted
• more problems correct
• Percent Correct: Problems correct/Problems attempted
• increase in Percent Correct
23
Practice-as-usualGrowth Mindset Message
Positive Message
3. Add motivational messages
Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!
24
Does any positive message work?
• Growth Mindset Message
• "Remember, the more you practice the smarter you become.”,
• "Mistakes help you learn. Think hard to learn from them.”
• Positive Message• "Some of these problems are
hard. Just do your best."• "This might be a tough problem,
but we know you can do it.”
• Practice-as-usual
25
Effects of Positive Messages?
• Positive Messages ~= Practice-as-Usual
• Growth Mindset > Positive• extra problems attempted • more problems correct • increase in Percent Correct
26
Computational Modeling
• Williams, Mitchell, Heffernan. MOOC Research Initiative grant from Gates Foundation & Athabasca. Investigating the benefits of embedding motivational messages in online exercises.
• 2 million users on 12 kinds of fractions exercises, ~100 problems each
• Moderators & Mediators• Item Response Theory• Non-parametric Bayesian clustering of Users (CrossCat, JMLR)• Model latent knowledge states
27
Synthesize Scientific Findings
• Williams, J.J. (2013)Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.
• www.josephjaywilliams.com/education
28
Experimental Paradigm: R.E.P.E.A.T.• Williams, J. J. (2013). Finding connections
between basic experimental research and realistic online education contexts. In J. J. Williams (chair), Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science.
• Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society. (pdf)
29
Experiments
• Williams, J.J. & Williams, B.A. (under review). Online A/B Tests & Experiments: A Practical But Scientifically Informed Introduction. Course proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. (pdf)
• Williams, J.J., Heffernan, N., & Koedinger, K. Experiments at Scale: Instrumenting MOOCs for experimentation and course-improving data analysis. Tutorial proposal submitted to the First Annual ACM Conference on Learning at Scale.
30
Experiment-Focused Design
• Williams, J.J. & Williams, B. A. (2013). Using Interventions to Improve Online Learning. Paper to be presented at the NIPS 2013 Workshop on Data Driven Education.
31
Review
• Explanation & Learning• Teaching Learning Strategies• Motivational Messages• Experimental Paradigm• Experiment-focused Design• Williams, J.J., Klemmer, S., Kizilcec, R., &
Russel, D. (under review). Learning Innovations at Scale. Workshop proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada.
32
Acknowledgements
• Jascha Sohl-Dickstein• Jace Kohlmeier & Khan Academy• Sam Maldonado• Lytics Lab (lytics.stanford.edu)• VPOL (Vice Provost of Online
Learning, online.stanford.edu)
Recommended