View
61
Download
0
Category
Preview:
Citation preview
1
The Learning Tracker
A Learner Dashboard that Encourages Self-Regulation in MOOC Learners
Ioana JivetSeptember 19th, 2016
2
Agenda
• Motivation
• Learning Tracker design
• Experimental setup
• Results
• Conclusion and outlook
3
Motivation
4
What is a MOOC?
Massive Open Online Course
Best Courses. Top Institutions. Learn anytime, anywhere.
• 35 million learners• 500 universities• 4 200 MOOCs
5
Dropout as a main challenge
• Low completion rates <15 % (Jordan, 2016)
• Underdeveloped learning skills and study habits– High autonomy– Role of the teacher– Low metacognitive awareness
6
Self Regulated Learning
• Definition: capability of the learner “to adjust her actions and goals to achieve desired results in light of changing environmental conditions”
(Zimmerman, 1990)
• Major success factor in online learning environments, including MOOCs
• Lack of learner support in current MOOC platforms
7
Learner support on MOOC platforms - edX
8
Aim
Investigate how self-regulated learning skills can be enhanced in MOOC learners
Encouraging metacognition and self-reflection on learning behaviour
Providing feedback through social comparison with successful learners on a learner dashboard
9
Learning Tracker design
10
Development
Design-based research methodology• Incremental• Evaluation on edX MOOCs offered by TU Delft
Two components• Data• Visualisation
First iteration Evaluation
January – March 2016
Second iteration Evaluation
April – June 2016
11
Behaviourmetrics
Data
edX trace logs Widget
Examples of metrics displayed on the widget• Number of graded quizzes attempted• Number of forum visits• Timeliness of assignment submission
Social comparison with the average graduate
12
6 Metrics
2 Information sets
13
Preliminary evaluation of the first iteration
• Metric configuration• Additional information set
– Average graduate in the following week– Reflection and planning support
Adjustments in the second iteration
14
Additional information set
15
Preliminary evaluation of the first iteration
• Metric configuration• Additional information set
– Average graduate at the end of current week– Reflection and planning support
• Interactive elements
Adjustments in the second iteration
16
Interactive elements – information sets
17
Interactive elements - tooltip
18
Implementation of the widget
edX trace logs
Behaviourmetrics
Widget
19
Experimental setup
20
Experimental setup
Three TU Delft MOOCs– Weekly publication of learning material– Video lectures, weekly assignments, practice
quizzes– Graduation: >60% final score
Replicated longitudinal study
21
Experimental setup
Method: randomized controlled trial– Demographic analysis to ensure populations
are sufficiently randomized
WaterX SewageX InnovationX
Test group 5 460 4 038 1 184
Control group 5 483 4 099 1 168
Total enrolled 10 943 8 137 2 352
22
Results
23
Learners’ performance
RQ1 Are learners more likely to complete the course when they can compare their behaviour to that of previous graduates?
24
Learners’ performance – graduation
Higher graduation rate.
25
Learners’ performance – final grades
More learners graduate, but they do not pursue higher grades.
26
Learners’ performance – final grades
More learners graduate, but they do not pursue higher grades.
27
Learners’ behaviour
RQ2.1 Do learners become more engaged with the MOOC when they can compare their behaviour with that of successful
learners?
28
Learners’ engagement – course material
Learners are more engaged with the graded course material.
WaterX SewageX Innovationx
Graded quizzes .036 .114 .044
Practice non-graded quizzes .512 .071 -
Mann-Whitney test results (p-values) between the test group and the control group.
– Significance level α = .050– Significant differences are marked in bold.
29
More learners are engaged with graded course content.
Learners’ engagement – course material
30
Learners’ engagement – course material
More learners are engaged with graded course content.
31
Learners’ self-regulation
RQ2.2 Do learners show improvement of their time-management skills when they
compare their behaviour to that of successful learners?
32
Learners’ self-regulation - procrastination
WaterX SewageX Innovationx
Timeliness(recommended)
.055 .113 .039
Timeliness(actual)
.040 .145 .035
Mann-Whitney test results (p-values) between the test group and the control group.
– Significance level α = .050– Significant differences are marked in bold.
Learners procrastinate less.
33
Learners’ self-regulation – procrastination
Learners procrastinate less.
34
Learners’ on-trackness
RQ2.3 Do learners change their behaviour so it becomes similar to that of successful learners when they compare themselves
to it?
35
Learners’ on-trackness
Similarity between a learners’ behaviour and that of the average graduate
1. Compute on-trackness score weekly2. Cluster learners based on the evolution
of the on-trackness score
36
Learners’ on-trackeness – clusters
No conclusive evidence that the Learning Tracker influences the distribution of learners into clusters.
on-track
behind, but keep up
behind, initial activity
behind, no activity
37
Conclusion and outlook
38
Overall outcomes
• Higher likelihood of graduation
• No immediate effect on self-regulated behaviour (e.g. procrastination)
• Limited feedback affects behaviour
• On-trackness classification based on behaviour similarity
39
Future work
Testing different definitions for success
Personalized feedback (demographics)
Social effects– behaviour uniformization– motivation
Extensive longitudinal evaluations
40
Demonstration paper
Davis, Chen, Jivet, Hauff, & Houben, 2016: Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback
– In Learning Analytics and Knowledge 2016 Learning Analytics for Learners Workshop.
41
Thank you!
42
edX FTP
Technical architecture
43
Learners’ on-trackness - score
Arithmetic weighted sum of metric deviations
44
Learners’ on-trackness - score
1. Behind
45
Learners’ on-trackness - score
1. Behind2. On-track
46
Learners’ on-trackness - score
1. Behind2. On-track3. Ahead
47
Inspiration – Search Dashboard
Impact of reflection and social comparison on search behavior
(Bateman, 2012)
Reference model – expert searchers
48
Behaviour metrics used on the widget
49
Behaviour metrics
Recommended