Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Evidence-based work on learning and instruction George Mason University June 6, 2012
2
• Integrating the design, building, monitoring, and improvement of learning environments; individualize learning experiences using our scale; and, ultimately, drive greater student career success.
• Former CLO for K12, Inc. – structured use of technology, cognitive science, on-line and off-line materials for 1,700 teachers, 55k students
• Former Publisher and General Manager for DK Multimedia, Inc. • Management consultant with McKinsey & Company • Education:
- Ph.D. in Electrical Engineering and Computer Science from MIT - M.D. from Harvard Medical School - M.A. in Electrical Engineering and Computer Science from MIT - M.A. in Mathematics from Oxford University - B.S. in Electrical Engineering and B.S. with Honors in Mathematics
from the University of Washington
Bror Saxberg Chief Learning Officer, Kaplan, Inc.
3
• Kaplan University • Kaplan Legal Education • Kaplan Professional
Education • Nursing • Kaplan Continuing Education • KNEXT
• KTPA • Kaplan Tutoring • Kaplan Bar Review • Kaplan Publishing
• Kaplan Higher Ed – Europe • Kaplan Professional – Europe
• Kaplan Higher Ed – Asia • Kaplan Professional – Asia • Kaplan Higher Ed – Australia • Kaplan Professional –
Australia • In Country Pathways – China • Franklyn Scholar • Carrick Education • Global Knowledge Solutions
Kaplan education spans domains and geography
Kaplan University Group
Kaplan Higher Education Campuses
Kaplan Test Prep
Kaplan Asia Pacific Kaplan United Kingdom • Kaplan Int’l Colleges
• Global Pathways
Kaplan International Colleges
4
What we’re trying to do
5
What we’re trying to do
6
What we’re trying to do
7
What we’re trying to do
8
What we’re trying to do
9
Agenda
• What evidence says about learning • What this means for the design of instruction • What happens when you do this for real
10
Agenda
• What evidence says about learning • What this means for the design of instruction • What happens when you do this for real
11
Much research to guide us
Learning Events
(hidden - inside students’ minds)
Student Performance
(observable -indicates
knowledge)
Instructional Events
(in the learning environment)
Knowledge
• Explicit: Information, Explanation, Examples, Demos
• Implicit: Practice tasks/activities (prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What • Implicit/Procedural/How • Knowledge Components
(Procedures + Facts, Concepts, Principles, Processes)
• Response accuracy/errors • Response fluency/speed • Number of trials • Amount of assistance (hints) • Reasoning
Koedinger, K.R., Corbe/, A.T., and Perfe5, C. (2010). The Knowledge-‐Learning-‐InstrucEon (KLI) Framework: Toward Bridging the Science-‐PracEce Chasm to Enhance Robust Student Learning (DraN manuscript from the Pi/sburgh Science of Learning Center)
12
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure
Sup
porti
ve/C
once
ptua
l Fact
Concept
Process
Principle
13
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure Sequence of decision and action steps to perform tasks; when and how to do things
• Prosecuting a criminal • Deciding if capital gains tax applies
Sup
porti
ve/C
once
ptua
l Fact
Concept
Process
Principle
14
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure Sequence of decision and action steps to perform tasks; when and how to do things
• Prosecuting a criminal • Deciding if capital gains tax applies
Sup
porti
ve/C
once
ptua
l Fact Isolated, unique piece of information; one instance
• 52 Grosvenor Place • 2+3=5
Concept
Process
Principle
15
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure Sequence of decision and action steps to perform tasks; when and how to do things
• Prosecuting a criminal • Deciding if capital gains tax applies
Sup
porti
ve/C
once
ptua
l Fact Isolated, unique piece of information; one instance
• 52 Grosvenor Place • 2+3=5
Concept Sets of items that share common attributes, common name; multiple examples
• Dog • Money • Happiness
Process
Principle
16
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure Sequence of decision and action steps to perform tasks; when and how to do things
• Prosecuting a criminal • Deciding if capital gains tax applies
Sup
porti
ve/C
once
ptua
l Fact Isolated, unique piece of information; one instance
• 52 Grosvenor Place • 2+3=5
Concept Sets of items that share common attributes, common name; multiple examples
• Dog • Money • Happiness
Process Flow of events or procedures; how things work
• Workflow • Chemical process
Principle
17
5 types of outcomes determine TYPE of information and practice
Knowledge Component Definition Example
Procedure Sequence of decision and action steps to perform tasks; when and how to do things
• Prosecuting a criminal • Deciding if capital gains tax applies
Sup
porti
ve/C
once
ptua
l Fact Isolated, unique piece of information; one instance
• 52 Grosvenor Place • 2+3=5
Concept Sets of items that share common attributes, common name; multiple examples
• Dog • Money • Happiness
Process Flow of events or procedures; how things work
• Workflow • Chemical process
Principle Guidelines, rules that govern, predict, explain events; relationships among concepts
• Supply and demand • 80/20 principle • Novices need structure
18
3 stages of learning determine instructional elements and sequence
Stage Characteristics Implications for Instructional Design
1.
2.
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, ExperEse: Its Structure and AcquisiEon.
Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007)
19
3 stages of learning determine instructional elements and sequence
Stage Characteristics Implications for Instructional Design
1. Declarative
• Knowledge “about”, “that”, “what” “why”;
• Can be stated verbally; • Conceptual network • Conscious
Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures
2.
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, ExperEse: Its Structure and AcquisiEon.
Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007)
20
3 stages of learning determine instructional elements and sequence
Stage Characteristics Implications for Instructional Design
1. Declarative
• Knowledge “about”, “that”, “what” “why”;
• Can be stated verbally; • Conceptual network • Conscious
Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures
2. Procedural
• Knowledge “how” • Sequence of “if-thens” • Potential to become unconscious
Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, ExperEse: Its Structure and AcquisiEon.
Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007)
21
3 stages of learning determine instructional elements and sequence
Stage Characteristics Implications for Instructional Design
1. Declarative
• Knowledge “about”, “that”, “what” “why”;
• Can be stated verbally; • Conceptual network • Conscious
Design clear, relevant, and accurate information displays, job aids, examples, reference material for all knowledge components: facts, concepts, principles, processes, procedures
2. Procedural
• Knowledge “how” • Sequence of “if-thens” • Potential to become unconscious
Design practice tasks to elicit student performance/ responses; monitoring systems to detect errors; and feedback/coaching to correct errors in performance
3. Automated
• Fluency • Expert • Unconscious • “10,000 hours”
Design opportunities for repeated frequent practice on the job and monitoring of speed and accuracy
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, ExperEse: Its Structure and AcquisiEon.
Fitts & Posner, (1967), John Anderson (2004, 2007); Anders Ericsson (2006, 2007)
22
Much research to guide us
Learning Events
(hidden - inside students’ minds)
Student Performance
(observable -indicates
knowledge)
Instructional Events
(in the learning environment)
Knowledge
• Explicit: Information, Explanation, Examples, Demos
• Implicit: Practice tasks/activities (prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What • Implicit/Procedural/How • Knowledge Components
(Procedures + Facts, Concepts, Principles, Processes)
• Response accuracy/errors • Response fluency/speed • Number of trials • Amount of assistance (hints) • Reasoning
Motivation
• Orientation/Inoculation • Monitoring • Diagnosis and treatment:
Persuasion, Modeling, Dissonance
• Value beliefs • Self-efficacy beliefs • Attribution beliefs • Mood/Emotion
• Behavior related to • Starting • Persisting • Mental Effort
• Self-reported beliefs
Koedinger, K.R., Corbe/, A.T., and Perfe5, C. (2010). The Knowledge-‐Learning-‐InstrucEon (KLI) Framework: Toward Bridging the Science-‐PracEce Chasm to Enhance Robust Student Learning (DraN manuscript from the Pi/sburgh Science of Learning Center)
23
4 beliefs influence mo/va/on
Sources: Bandura; Eccles & Wigfield; Pintrich & Schunk; Clark; Dweck
Beliefs
• Value • Self-Efficacy • Attribution • Mood
Motivated Behavior
• Starting • Persisting • Mental Effort
Learning/ Performance
• Practice • Test
Self-Efficacy
Effo
rt
High Moderate Low
Motivation Low High
Performance High Low
• Design materials and interaction to foster positive mood, high perception of value, moderate confidence, and attribution of success and failure to effort
• Design system for monitoring and guidance (group and individual)
24
Much research to guide us
Learning Events
(hidden - inside students’ minds)
Student Performance
(observable -indicates
knowledge)
Instructional Events
(in the learning environment)
Knowledge
• Explicit: Information, Explanation, Examples, Demos
• Implicit: Practice tasks/activities (prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What • Implicit/Procedural/How • Knowledge Components
(Procedures + Facts, Concepts, Principles, Processes)
• Response accuracy/errors • Response fluency/speed • Number of trials • Amount of assistance (hints) • Reasoning
Motivation
• Orientation/Inoculation • Monitoring • Diagnosis and treatment:
Persuasion, Modeling, Dissonance
• Value beliefs • Self-efficacy beliefs • Attribution beliefs • Mood/Emotion
• Behavior related to • Starting • Persisting • Mental Effort
• Self-reported beliefs
Metacognition • Structure • Guidance
• Planning, Monitoring • Selecting, Connecting
• Amount of guidance required/requested
Koedinger, K.R., Corbe/, A.T., and Perfe5, C. (2010). The Knowledge-‐Learning-‐InstrucEon (KLI) Framework: Toward Bridging the Science-‐PracEce Chasm to Enhance Robust Student Learning (DraN manuscript from the Pi/sburgh Science of Learning Center)
25
Agenda
• What evidence says about learning • What this means for the design of instruction • What happens when you do this for real
26
Instructional design: “Engineering” from learning science
Overviews Information Examples Practice Assessment Learning Outcomes
Motivational Guidance
Design
Deliver
Learning science strongly suggests an order to design and delivery
Clark, R.E., & Feldon, D. F. (2008). GEL (Guided Experiential Learning), Adaptable Expertise and Transfer of Training. Kirscher, P.A., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41, 75-86.
Knowledge Integration
27
Evidence-based instructional principles Accumulation of results from lab studies support: • Structure and guidance for novices
(Kirschner, Sweller, & Clark, 2006) • Demonstrations and worked examples
(Paas & van Merrienboer, 1994; Sweller, 2006) • Practice and corrective feedback
(Mathan & Koedinger, 2005) • Prompted self-explanation
(Aleven & Koedinger, 2002) • Multimedia use that minimizes extraneous cognitive load
(Mayer, 2009) • Targeting beliefs (value, confidence, and attributions) and
emotions (positive feelings) to influence motivation (Clark, 2004; Um et al., 2011)
28
Task-centered instruction
• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim • Teach everything needed for each task • Fade coaching/support over time
29
Knowledge Component
Presentation (Prepare) Practice/Assessment (Practice, Perform)
Info Example Remember Proxy for Remember Use** Proxy for Use **
Procedure
When to use; List of action and decision steps
Demonstration of when and how to perform
Recall when to use; Recall action and decision steps
Reorder steps; Recall next or missing steps
Decide when to use; Perform the steps (actions and decisions)
Critique performance or output of actions and decisions
Sup
porti
ve K
now
ledg
e
Fact * Statement of fact Statement of fact Recall fact Recognize fact when
presented with distractors Recall fact in task context
Concepts List of defining attributes
Examples; Non-examples
List defining attributes verbally or in writing
Recognize defining attributes when presented with distractors
Classify, identify or generate examples and non-examples
Critique someone else’s identification or generation of examples
Process/ System
List of phases, events and causes at each phase
Examples; simulations of phases, events, and causes
Recall phases, events, and causes
Recognize phases, events, and causes; Recall missing phases, events, and causes
Identify causes of faults in a process; Predict events in a process
Critique someone else’s description of causes or prediction of events in a process
Principle (cause and effect relationship)
Statement of cause and effect relationship
Examples, demonstration, simulation of cause and effect relationship
Recall the principle
Recognize the principle; Recall missing elements of the principle
Decide if principle applies; Predict an effect; Apply principle to solve a problem, explain a phenomenon or make a decision
Critique someone else’s application of the principle to solve a problem, explain a phenomenon or make a decision
Knowledge Integration
Explain the interconnections among conceptual knowledge components, or the conceptual foundation of procedures, or the procedural implementation of conceptual knowledge components
Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or select their reasoning about interconnections among knowledge components, for example the principle(s) that justify the application of a procedure.
Knowledge Transfer
Multiple and varied contexts for examples
Multiple and varied contexts for practice and assessment. Opportunities for students to explain how they would use the knowledge in other contexts
*Facts are concepts with single instances ** All Use and Proxy for Use Activities develop/require procedural knowledge
Presentation and practice match objectives (knowledge components)
30
Agenda
• What evidence says about learning • What this means for the design of instruction • What happens when you do this for real
31
ID can change instructional outcomes at scale
Principle Description Effect size (s.d. units)
Multimedia Use relevant graphics and text to communicate content 1.5
Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating integrated information
1.1
Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3
Modality Include audio narration where possible to explain graphic presentation 1.0
Redundancy Do not present words as both on-screen text and narration when graphics are present
.7
Personalization Script audio in a conversational style using first and second person 1.3
Segmenting Break content down into small topic chunks that can be accessed t the learner’s preferred rate
1.0
Pre-training Teach important concepts and facts prior to procedures or processes 1.3
Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc.
??
Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008
32
Impact is not small!
50% 1 sd
84%!
33
Redeveloping courses at scale
Read, Write, Discuss • Outcomes and content not
precisely aligned • Limited demonstrations, worked
examples, and practice • General assessment rubrics • High reliance on discussion boards
Existing courses
Prepare, Practice, Perform • Outcomes and content aligned • One lesson per objective • Demonstrations and worked examples • Practice, feedback before assessment • Detailed scoring guides • Less discussion/more practice • Standard instructor materials • Monitoring and support for motivation
Redesigned courses
34
Content Design
Items Prepare Practice Perform
Seminar Discussion Lessons Sets Overview
Course Level Outcome 1
Unit Outcome 1
Unit Outcome 2
Unit Outcome 3
Prepare 1
Practice 1
Perform 1
Prepare 2
Practice 2
Perform 2
Prepare 3
Practice 3
Perform 3
Lesson 1 Lesson 2 Lesson 3
Navigation
35
Narrated demonstrations
36
Explanation and demonstration of concepts
37
Practice with hints and feedback
38
Scenario-based practice
39
Prompted self-explanation
40
Motivation surveys
41
5
Student Preparation and Questions
Prior to attending the live seminar, or reviewing the seminar recording, complete the following tasks:
1. Check your grades and think about your performance in the previous unit:
• Where did you do well and where did you struggle? • What would help you do even better in the current unit?
2. Read the current unit Overview, Seminar, and Discussion pages.
3. Read the current unit recommended textbook pages.
4. Write down at least two questions you would like answers to in the seminar. CHAT: Type one of your questions into the chat window.
6
Overview
Seminar
Discussion Journal: A situation in your life where the guidelines for improving nonverbal communication could guard against misinterpretation.
Lesson 1 Identify verbal and nonverbal elements in personal and professional situations 1-2 hrs.
Lesson 2 Identify nonverbal communication principles in personal and professional situations 1-2 hrs.
Lesson 3 Explain instances of effective and ineffective communication in terms of how verbal and nonverbal elements work together 2-4 hrs.
UNIT 5 Review: What is Nonverbal Communication?
2. Identify nonverbal communication principles in personal and professional situations
15
UNIT 5 Review: Lesson 2 Practice
http://www.youtube.com/watch?v=bg0kSIJZiRQ&feature=related
PART 1: Which nonverbal communication principle is predominant in the woman’s reactions to her blind date?
Watch Item 2 video:
21
Overview (including Survey)
Seminar Discussion
Explain how improving your listening skills can increase the effectiveness of your communication in the workplace and in your personal life.
Lesson 1 Identify forms of nonlistening in personal and professional situations 1-2 hrs.
Lesson 2 Apply the principles of mindful listening to improve the effectiveness of communication in personal and professional situations 1-2 hrs.
UNIT 6 Preview: How Does Listening Enhance Our I.C.?
25
UNIT 6: Lesson 2 Practice
Watch the Online Dating video. Answer the three questions, referring to the scoring guide.
Online Dating
After answering a question, study the “Compare with Expert” response.
1. From the interaction does it seem to you that Chris’s mom is actively listening during the first third of this conversation? Why or why not?
2. Apply the principles of mindful listening to improve communication effectiveness
27
• The Practice and Perform activities are open book and untimed. Consult the textbook and online information in a separate browser window or tab as you do them.
• If you don’t understand the feedback in a Practice item, ask me about it and I will try to explain it.
• To do well on the Lesson 2 Perform activity, reference the scoring guide and the Prepare and Practice example responses for each question.
• Use the transcripts of the Lesson 2 video scenarios to help you answer the questions:
- For the Practice, see Online Dating: p. 166
- For the Perform, see Alan O’Connor: p. 217
UNIT 6: Tips for Success
29
Agenda Minutes Opening 5 ! Student Questions 10 ! Review Unit 5 10 ! View Unit 6 25 ! Preview Unit 7 5 ! Wrap Up 5 !
Poll Question
How many of you still have questions?
Post your questions in �Course Questions��
discussion board
! Yes ! No
(Link in Course Home menu)
Wrap Up
Instructor seminar materials more standardized and aligned with online content
42
= Unit 1 = Unit 3 = Unit 6
Provide materials to monitor and support motivation
43
What happened? Research design: quasi-experimental
Control Pilot Control Pilot Control PilotInterpersonal Communications 8 7 4 3 237 199Principles of Nutrition 6 4 3 2 148 89Medical Terminology 6 7 4 2 197 220
Total 20 18 11 7 582 508
n sections n instructors n studentsCourse
• 1,090 students (508 pilot; 582 control) • 87% female, average age 32; average household income $20,000 • 3 courses • 18 instructors • 20 sections (assigned to pilot or control) • 2 terms (Aug – Dec 2011)
44
Analysis
• Logistic regression to examine effect of course design on student success
• Success (1 or 0): Defined as: Pass (1 or 0) + Master course objectives (>=4 on 0-5 scale) + Stay (1 or 0)
• Controlled for variation in • Instructor prior student success rates • Student background variables
• Age, prior education, prior GPA, tenure, household income
• Calendar-based success variation
45
Student success: results controlling for variables • 11% higher
success rate
• 28% increase
• Students in redesigned courses were 1.6 times more likely to be successful
Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.
39%
50%
20%
30%
40%
50%
60%
70%
Control Pilot Adj
uste
d st
uden
t suc
cess
rate
Adjusted student success rates with 95% confidence limits
46
More to do: adjusted results varied by course
• Odds of success more than doubled (2.5 times more) in Interpersonal Communications
• 9% difference in Principles of Nutrition (not stat significant due to smaller sample) • Small improvement in Medical Terminology due to difficulty level of early units
33%#45%# 39%#
55%# 54%#42%#
0%#10%#20%#30%#40%#50%#60%#70%#
Interpersonal##Communica;ons#
Principles##of#Nutri;on#
Medical##Terminology#
Adjusted
(stude
nt(success(ra
te(
Error#bars:#+/I#1.00#SE#
Adjusted(student(success(rates(by(course(
Control# Pilot#
47
Student quote on benefits of added practice
“Something I found to be interesting was the degree of understanding between me and another individual that wasn’t in this class. A girl I had met in a previous term that has a similar degree plan but ended up in a regular medical terminology course, still we would discuss the differences and similarities between are assigned classes. During our unit 8 test she called me hysterical about all the different elements of the final tests and couldn’t seem to grasp the concept of the 1st part of the test i.e., analysis diagram, creating new terms from word roots etc. I was mystified that something that had become 2nd nature to me mainly due to the time spent every week filling out the Analysis Tables was so difficult for her to comprehend. It was at that point I realized all the griping I had done was actually the reason my level of understanding is more evolved than somebody who never experienced it.”
48
Quote from a student who previously failed
“This course was difficult for me to do. I tried to do this course when I attended another school and I failed it. I think the way the course was set up and how it broke everything down really helped me to understand it and pass it this time. I would not change a thing about how this course was set up.”
49
Satisfaction (end of term survey, 5 point scale)
Instructor satisfaction • Higher in redesigned courses (mean 4.6 vs 4.1) Why?
• Detailed scoring guides for assignments • Less time in discussions – more time to monitor and
communicate with at-risk students • Standard seminar format and content • Student materials: structure, clarity, practice
Student satisfaction
• Lower on end of course survey in redesigned courses (mean 4.4 vs 4.8), but still greater than 4 on 5 point scale. High positivity scores in motivation survey.
Why? • Courses more rigorous, more work to complete; this is a
common finding in other research (e.g., Clark, 1982)
50
What we’re trying to do
51
Useful references:
• Why Students Don’t Like School, Daniel Willingham – highly readable! ;-) • Talent is Overrated, Geoffrey Colvin – highly readable! ;-) • E-Learning and the Science of Instruction, Clark and Mayer, 2nd ed. • “First Principles of Learning,” Merrill, D., in Reigeluth, C. M. & Carr, A. (Eds.),
Instructional Design Theories and Models III, 2009. • How People Learn, John Bransford et al, eds. • “Design factors for educationally effective animations and simulations,” Plass,
J.L., Homer, B.D., Hayward, E.O., J Comput High Educ (2009) 21:31–61 • “The Implications of Research on Expertise for Curriculum and Pedagogy”,
David Feldon, Education Psychology Review (2007) 19:91–110 • “Cognitive Task Analysis,” Clark, R.E., Feldon, D., van Merrienboer, J., Yates,
K., and Early, S.. in Spector, J.M., Merrill, M.D., van Merrienboer, J. J. G., & Driscoll, M. P. (Eds.), Handbook of research on educational communciatinos and technology (3rd ed., 2007) Lawrence Erlbaum Associates