1

The Learning Tracker

A Learner Dashboard that Encourages Self-Regulation in MOOC Learners

Ioana JivetSeptember 19th, 2016

2

Agenda

• Motivation

• Learning Tracker design

• Experimental setup

• Results

• Conclusion and outlook

3

Motivation

4

What is a MOOC?

Massive Open Online Course

Best Courses. Top Institutions. Learn anytime, anywhere.

• 35 million learners• 500 universities• 4 200 MOOCs

5

Dropout as a main challenge

• Low completion rates <15 % (Jordan, 2016)

• Underdeveloped learning skills and study habits– High autonomy– Role of the teacher– Low metacognitive awareness

6

Self Regulated Learning

• Definition: capability of the learner “to adjust her actions and goals to achieve desired results in light of changing environmental conditions”

(Zimmerman, 1990)

• Major success factor in online learning environments, including MOOCs

• Lack of learner support in current MOOC platforms

7

Learner support on MOOC platforms - edX

8

Aim

Investigate how self-regulated learning skills can be enhanced in MOOC learners

Encouraging metacognition and self-reflection on learning behaviour

Providing feedback through social comparison with successful learners on a learner dashboard

9

Learning Tracker design

10

Development

Design-based research methodology• Incremental• Evaluation on edX MOOCs offered by TU Delft

Two components• Data• Visualisation

First iteration Evaluation

January – March 2016

Second iteration Evaluation

April – June 2016

11

Behaviourmetrics

Data

edX trace logs Widget

Examples of metrics displayed on the widget• Number of graded quizzes attempted• Number of forum visits• Timeliness of assignment submission

Social comparison with the average graduate

12

6 Metrics

2 Information sets

13

Preliminary evaluation of the first iteration

• Metric configuration• Additional information set

– Average graduate in the following week– Reflection and planning support

Adjustments in the second iteration

14

Additional information set

15

Preliminary evaluation of the first iteration

• Metric configuration• Additional information set

– Average graduate at the end of current week– Reflection and planning support

• Interactive elements

Adjustments in the second iteration

16

Interactive elements – information sets

17

Interactive elements - tooltip

18

Implementation of the widget

edX trace logs

Behaviourmetrics

Widget

19

Experimental setup

20

Experimental setup

Three TU Delft MOOCs– Weekly publication of learning material– Video lectures, weekly assignments, practice

quizzes– Graduation: >60% final score

Replicated longitudinal study

21

Experimental setup

Method: randomized controlled trial– Demographic analysis to ensure populations

are sufficiently randomized

WaterX SewageX InnovationX

Test group 5 460 4 038 1 184

Control group 5 483 4 099 1 168

Total enrolled 10 943 8 137 2 352

22

Results

23

Learners’ performance

RQ1 Are learners more likely to complete the course when they can compare their behaviour to that of previous graduates?

24

Learners’ performance – graduation

Higher graduation rate.

25

Learners’ performance – final grades

More learners graduate, but they do not pursue higher grades.

26

Learners’ performance – final grades

More learners graduate, but they do not pursue higher grades.

27

Learners’ behaviour

RQ2.1 Do learners become more engaged with the MOOC when they can compare their behaviour with that of successful

learners?

28

Learners’ engagement – course material

Learners are more engaged with the graded course material.

WaterX SewageX Innovationx

Graded quizzes .036 .114 .044

Practice non-graded quizzes .512 .071 -

Mann-Whitney test results (p-values) between the test group and the control group.

– Significance level α = .050– Significant differences are marked in bold.

29

More learners are engaged with graded course content.

Learners’ engagement – course material

30

Learners’ engagement – course material

More learners are engaged with graded course content.

31

Learners’ self-regulation

RQ2.2 Do learners show improvement of their time-management skills when they

compare their behaviour to that of successful learners?

32

Learners’ self-regulation - procrastination

WaterX SewageX Innovationx

Timeliness(recommended)

.055 .113 .039

Timeliness(actual)

.040 .145 .035

Mann-Whitney test results (p-values) between the test group and the control group.

– Significance level α = .050– Significant differences are marked in bold.

Learners procrastinate less.

33

Learners’ self-regulation – procrastination

Learners procrastinate less.

34

Learners’ on-trackness

RQ2.3 Do learners change their behaviour so it becomes similar to that of successful learners when they compare themselves

to it?

35

Learners’ on-trackness

Similarity between a learners’ behaviour and that of the average graduate

1. Compute on-trackness score weekly2. Cluster learners based on the evolution

of the on-trackness score

36

Learners’ on-trackeness – clusters

No conclusive evidence that the Learning Tracker influences the distribution of learners into clusters.

on-track

behind, but keep up

behind, initial activity

behind, no activity

37

Conclusion and outlook

38

Overall outcomes

• Higher likelihood of graduation

• No immediate effect on self-regulated behaviour (e.g. procrastination)

• Limited feedback affects behaviour

• On-trackness classification based on behaviour similarity

39

Future work

Testing different definitions for success

Personalized feedback (demographics)

Social effects– behaviour uniformization– motivation

Extensive longitudinal evaluations

40

Demonstration paper

Davis, Chen, Jivet, Hauff, & Houben, 2016: Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback 

– In Learning Analytics and Knowledge 2016 Learning Analytics for Learners Workshop.

41

Thank you!

42

edX FTP

Technical architecture

43

Learners’ on-trackness - score

Arithmetic weighted sum of metric deviations

44

Learners’ on-trackness - score

1. Behind

45

Learners’ on-trackness - score

1. Behind2. On-track

46

Learners’ on-trackness - score

1. Behind2. On-track3. Ahead

47

Inspiration – Search Dashboard

Impact of reflection and social comparison on search behavior

(Bateman, 2012)

Reference model – expert searchers

48

Behaviour metrics used on the widget

49

Behaviour metrics