Measuring Engagement:Learning Analytics in Online Learning

Dr. Griff RichardsThompson Rivers UniversityOpen Learning

Electronic Kazan, 19 April 2011

60oN

49oN

Distance LearningIn Canada

- Is growing, especially in K-12 public schools

-15% of 2008 high school graduates in BC took at least one on-line course

-Increase in “Blended Learning”, online activities for F2F courses.

Kamloops, British Columbia

Learning Engagement

The more a learner interacts with the content and with their peers about the content, the more they are likely to internalize it, and remember it...

Richard Snow, 1980

Interaction Equivalency Theory(Garrison & Anderson)

The quality of the interaction is more important than the source

CourseContent

Instructor

Other learners

Learning Engagement (Kuh)

Learning engagement promotes student retention and academic success.

(George Kuh)

Engagement can be broadly measured for a campus using National Engagement School Survey Instrument (NESSI).

Learning Engagement (Kuh)

Five Factors:1. active and collaborative learning,

2. student-faculty interaction, 3. supportive campus environment, 4. enriching educational experiences, and 5. the level of academic challenge.

Engagement Online

on line

Measuring Engagement• Engagement is “action based on internal

motivation”• Can not directly measure engagement• Must look for external traces, actions that

indicate interest, and involvement

Kuh – NESSI – sample every year of 100,000s of students -> statistical trends, but not individual

as results come too late to correct problems!

Measuring Online Engagement

Engagement is inferred from activity.• Student interaction with Online systems

leaves an “exhaust trail” of their activity• Learning Analytics is not a statistical sample,

it is all the data for all learners• Questions: What patterns to look for? and How to interpret them?

• LMS interaction data e.g. Moodle discussion

• Extract linked list

• Plot interactions in star map

Example: Snapp

SNAPP Visualization

SNAPP shot of Welcome

Students “engaged”

SNAPP shot of Welcome

Individuals with lower engagement, 3 or less messages

Does Activity = Engagement ?

Beer (2010) plotted total LMS interactions with academic grades. Does activity = engagement?

Is Blackboard more engaging than Moodle?

Activity Outside LMS

• As learners become engaged, they move to more engaging channels: email, Elluminate, Skype

• This activity is not tracked by LMS.

• No data available.

Interpretation of Analytics

• Data patterns require verification• Quantitative data requires interpretation--> make and test hypotheses--> create useful models• When we measure something we risk

changing it.– e.g. If learners know we count hits they may

make meaningless hits to fool the system.

Analytics for Learners!

• The same analytics should be able to provide easy to understand information dashboards for students.

How can analytics improve online learning?

• Measuring something is the first step• “Better to measure something than to

measure nothing” (Scrivens)

• Need more data than page hits. We also need to ask learners about their experience, what worked, what needs improvement.

Analytics can help build Quality Systems

Multi-layered system, Timely, Qualitative & Quantitative data

Steady enrolments

Credible learning

Happylearners

Strategic course redesign

Dynamic Evaluation Model (Richards & Devries)

» Preparation Conduct Reflection

• Design

• Facilitation

• Learning

Analytics at the activity level

Dynamic Evaluation Model

» Preparation Conduct Reflection

• Design

• Facilitation

• Learning

Timely feedback enables quick fixes

• Too little

• Too much

• Just right

Masha & Medved

We need to find balance.

Thank you!

Questions?

Measuring Engagement:Learning Analytics in Online Learning

Dr. Griff RichardsThompson Rivers UniversityOpen Learning

Electronic Kazan, 19 April 2011