Embed Size (px)
Citation preview
Improving Learning from Web-Improving Learning from Web-Based Training Courses: Research Based Training Courses: Research
EvidenceEvidence
Dr. Traci SitzmannAdvanced Distributed Learning
Co-Laboratory
2
Web-based Learning
EmbeddedTraining
OnlineGaming
DistributedSimulation
ContentRepositories
Office
School
Home Field
InTransit
Realizing the ADL Vision
ARE WE: Providing access to the highest quality education and training, tailored to individual needs, delivered cost effectively,
anywhere and anytime?
ARE WE: Providing access to the highest quality education and training, tailored to individual needs, delivered cost effectively,
anywhere and anytime?
ADL Co-Lab Network - U.S.►ADL Co-Laboratory Hub
Alexandria, Virginia
►Joint ADL Co-LabOrlando, Florida
►Academic ADL Co-LabMadison, Wisconsin
►Workforce ADL Co-Lab Memphis, Tennessee
Canada ADL Partnership Lab
(Ottawa - DND)
United Kingdom ADL Partnership Lab
(Telford, England)
NATO/PfP
(26 + 20)
Latin America ADL Partership Lab
ILCE – (13 Latin American countries)
KOR Ministry of Commerce, Industry, & EnergyTWN Ministry of Economic AffairsSGP Singapore Armed ForcesTHAI Ministry of Science and Technology
DEST Department of Education, Science and Training
ILCE Instituto Latinoamericano de la Comunicación Educativa
PfP Partnership for Peace/ADLWorking Group
ADL Global Partnerships
NATO NATO Training Group
In negotiation
Taiwan
Singapore
Australia ADL Partnership Lab
(DEST)
KoreaADL Partnership Lab
(MOCIE)
5
Overview
► Meta-analysis on the effectiveness of Web-based instruction (WBI)
► Prompting self-regulation to improve learning outcomes during WBI
6
7
Purpose of Web-based Training Meta-analysis
►Compare Web-based instruction (WBI) and blended learning (BL) to CI in terms of:● Their effectiveness for teaching declarative and
procedural knowledge
● Student reactions to training
►Examine training design characteristics which influence learning from WBI
8
Participant Demographics
►19,331 trainees►168 training courses
● 113 undergraduate courses● 29 graduate courses● 26 corporate training courses
►Average age = 24 years
9
Analyses
►Hedges and Olkin’s (1985) procedure → mean corrected d effectd > 0 indicates WBI is more effective than CI
d = 0 indicates WBI and CI are equally effective
d < 0 indicates CI is more effective than WBI
►Subgroup procedure → test for categorical moderators
►Sample size weighted correlations & hierarchical regression analysis → test for continuous moderators
10
Meta-Analytic Notional Example
Research report
Sample size
d Total
Arbaugh 150 .05 d = .22
Jackson 288 .26
Manning 35 -.02
Smith 71 .18
11
Effectiveness of Web-Based Instruction
Declarative knowledge d = .15*● WBI is 6% more effective than CI for teaching
declarative knowledge
Procedural knowledge d = - .07● WBI and CI are equally effective for teaching
procedural knowledge
Satisfaction with training d = .00● Trainees are equally satisfied with WBI and CI
12
Effectiveness of Blended Learning Courses
Declarative knowledge d = .34*● BL is 13% more effective than CI for teaching
declarative knowledge
Procedural knowledge d = .52*● BL is 20% more effective than CI for teaching
procedural knowledge
Satisfaction with training d = - .15*● Trainees are 6% more satisfied with CI than BL
13
Effect of Trainee & Situational Characteristics
►Is the effectiveness of WBI, relative to CI, for teaching declarative knowledge influenced by the following factors?● Instructional methods● Experimental design● Learner control● Human interaction● Practice● Feedback● Age and population (students vs. employees)
14
Meta-Analytic Notional Example
Research report
Sample size
d Learner control
Arbaugh 150 .05 Low
Manning 35 -.02 Low
Jackson 288 .26 High
Smith 71 .18 High
15
Richard Clark’s Theory
Media are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes change in our nutrition.” -- Clark (1983)● Instructional methods include lecture, practice,
examples, discussion, videos, etc.
● Delivery media include CI, computer-assisted instruction, WBI, etc.
16
Similarity of Instructional Methods
Hypothesis: WBI and CI will be equally effective for teaching declarative knowledge when similar instructional methods are used to deliver the two courses.● Same instructional methods d = .04
● Different instructional methods d = .29
►Instructional methods rather than delivery media predict learning outcomes
17
Experimental Design
Hypothesis: WBI and CI will be equally effective for teaching declarative knowledge when studies used a true experimental design.● Experimental designs d = -.26
● Quasi-experimental designs d = .18
►Results support Clark’s position that media effects in single study research are largely spurious
18
19
Learner Control
Learner control refers to the extent to which trainees have control over the content, sequence, and pace of instruction● Low learner control d = .07● High learner controld = .30
►Trainees learn more when with a high than a low level of learner control during WBI
20
Human Interaction
Human interaction indicates the extent to which trainees interact with the instructor and other trainees throughout the course● Low human interaction d = .19
● High human interaction d = .18
21
Practice & Feedback
► Practice results● Only WBI incorporated practice d
= .31● WBI and CI incorporated practice d = .16● Only CI incorporated practice d = -.27● Neither WBI or CI incorporated practice d =
-.25
► Feedback results● Only WBI incorporated feedback d
= .33● WBI and CI incorporated feedback d = .16● Only CI incorporated feedback d = -.27● Neither WBI or CI incorporated feedback d = .08
22
Population & Age
Does learning from WBI, relative to CI, differ for students and employees after controlling for the age of trainees?
► Age accounted for a significant 42.2% of the variance in declarative knowledge
► Effectiveness of the WBI did not differ for students and employees after controlling for the age of trainees
23
Designing Effective Web-Based Courses
• Incorporate a variety of instructional methods• Require active involvement of trainees• Provide an Internet skills course for trainees lacking
technical skills• Provide trainees with control during training• Online communication should be synchronous rather
than asynchronous• Trainees should practice the training material• Provide trainees with feedback throughout the course
24
25
Self-Regulation
►Self-regulation is a process that enables an individual to guide his or her goal-directed activities over time and across changing circumstances (Karoly, 1993)
►Gradual effect on learning
26
Self-Regulation Prompts
Am I concentrating on learning the training material?
Do I need to continue to review before taking the final exam?
27
Self-Regulation Prompts
Am I setting learning goals to ensure I have a thorough understanding of the
training material?
Do I understand all of the key points of the training material?
28
Self-Regulation Conditions
►Immediate self-regulation● Prompted self-regulation after all 10 modules
►Delayed self-regulation● Prompted self-regulation after modules 5-10
►No self-regulation● Did not prompt self-regulation
29
Study 1
►Online Blackboard training course►Learning was assessed with multiple-choice and
performance exams at the end of training►Participants:
● 93 trainees
● 90% were instructors at a university or community college
● 78% had a master’s degree, Ph.D., or M.D.
● Average age = 44 years
30
Study 2
►Air Traffic Controller Task►Assessed performance in a simulation►Examined two dimensions of performance:
● Basic
● Strategic
►Participants ● 171 undergraduates
● Average age = 19 years
Results
-0.1
-0.05
0
0.05
0.1
0.15
0.2
1 2 3 4 5 6 7 8 9 10
Module
PerformanceControl
Results
-0.1
-0.05
0
0.05
0.1
0.15
0.2
1 2 3 4 5 6 7 8 9 10
Module
Performance
Immediateself-regulation
Control
Results
-0.1
-0.05
0
0.05
0.1
0.15
0.2
1 2 3 4 5 6 7 8 9 10
Module
Performance
Immediateself-regulation
Delayedself-regulation
Control
34
All Trainees Benefit from Prompts?
►Cognitive ability ● High ability trainees may be more capable of
managing their own learning
►Self-efficacy -- trainees’ perceptions of their ability to reach a goal● Trainees with high self-efficacy are more resilient
when they receive negative feedback
● Trainees with high self-efficacy may be more likely to use the prompts to adjust their training behavior
Results
-1
-0.5
0
0.5
1
1.5
1 2 3 4 5 6 7 8 9
Trial
Strategic Performance
Immediateself-regulation,high ability
Delayed self-regulation,high ability
Control, highability
Results
-1
-0.5
0
0.5
1
1.5
1 2 3 4 5 6 7 8 9
Trial
Strategic Performance
Immediateself-regulation,low ability
Delayed self-regulation, lowability
Control, lowability
Results
-1
-0.5
0
0.5
1
1.5
1 2 3 4 5 6 7 8 9
Trial
Strategic Performance
Immediateself-regulation,high ability
Immediateself-regulation,low ability
Delayed self-regulation,high ability
Delayed self-regulation, lowability
Control, highability
Control, lowability
Results
-1.5
-1
-0.5
0
0.5
1
1.5
2
1 2 3 4 5 6 7 8 9
Trial
Performance
Immediate self-regulation, highself-efficacy
Delayed self-regulation, highself-efficacy
Control, highself-efficacy
Results
-1.5
-1
-0.5
0
0.5
1
1.5
2
1 2 3 4 5 6 7 8 9
Trial
Performance
Immediate self-regulation, lowself-efficacy
Delayed self-regulation, lowself-efficacy
Control, low self-efficacy
Results
-1.5
-1
-0.5
0
0.5
1
1.5
2
1 2 3 4 5 6 7 8 9
Trial
Performance
Immediate self-regulation, highself-efficacy
Immediate self-regulation, lowself-efficacy
Delayed self-regulation, highself-efficacy
Delayed self-regulation, lowself-efficacy
Control, highself-efficacy
Control, low self-efficacy
41
Conclusions
►Prompting self-regulation improved learning outcomes
►Trainees with high self-efficacy and high ability may benefit more from the prompts
►Prompting self-regulation is a no-cost intervention ►Can be reused in any web-based training course
This is a good time to tell you that research has shown that asking yourself questions about whether you are concentrating on learning the training material will increase your performance on the test following training. The training program will periodically ask you questions about where you are directing your mental resources and whether you are making progress towards learning the training material. Your goal is to insert a goal relevant to the course content. Think about these questions and use your responses to decide how to allocate your review time.
Am I concentrating on learning the training material?
Are there areas of training I am going to have a difficult time remembering for the final exam?
44
Are the study tactics I have been using effective for learning the training material?
Do I know more about the training material than when training began?
45
Do I have thoughts unrelated to training that interfere with my ability to focus on training?
Have I forgotten some of the ideas introduced in previous training material?
46
Research indicates that asking yourself these questions throughout training results in higher levels of performance in training. You should continue to ask yourself these questions as you study the training material.
Am I setting learning goals to ensure that I will be ready to take the posttest?
Would I do better on the final exam if I studied more?
47
Have I developed a strategy for increasing my knowledge of the training material?
Am I making progress towards insert goal relevant to the training content?
48
Am I distracted during training?
Have I reviewed the training material as much as necessary for the final exam?
49
Are the study strategies I'm using helping me learn the training material?
Have I spent enough time reviewing to remember the information for the final exam?
50
Am I setting learning goals to help me perform better on the final exam?
Do I need to continue to review before taking the final exam?
51
Am I focusing my mental effort on the training material?
Do I know enough about the training material to reach my goal of insert goal relevant to the
course content?
