Performance Analysis for Blended MOOCs on IITBombayX...Performance Analysis for Blended MOOCs on IITBombayX Rahul Dev Parashar Guided by: Prof. Deepak B Phatak Indian Institute of

Performance Analysis for Blended MOOCs onIITBombayX

Rahul Dev Parashar

Guided by: Prof. Deepak B Phatak

Indian Institute of Technology, Bombay

October 14, 2015

Rahul Dev Parashar (IIT Bombay) Stage 1 presentation October 14, 2015 1 / 35

Overview

1 IntroductionPedagogical DriftChallenges in blended learningWhat is performance analysis?Problem Statement

2 IITBombayX ArchitectureVarious ComponentsData Modules

3 Proposed Approach and PrototypeProposed ApproachPrototypeData Analytics

4 ObservationsInvalidated JSON recordsUnclassified events/fields

5 Future WorkRahul Dev Parashar (IIT Bombay) Stage 1 presentation October 14, 2015 2 / 35

Introduction

Introduction


Introduction Pedagogical Drift

Pedagogical Drift

Various methodologies

Traditional learning

MOOCs (Massive Open Online Courses)

Blended MOOCs



Pedagogical Drift




Blended MOOCs



Pedagogical Drift




Blended MOOCs



Traditional Learning

Advantages

Learning through interaction with other students

Teachers can motivate students

Disadvantages

Difficult to handle large class size

Learning habit of students can be different

Learning may be hindered by the ability of teacher to teach thesubject



Traditional Learning

Advantages

Learning through interaction with other students

Teachers can motivate students

Disadvantages


Learning habit of students can be different

Learning may be hindered by the ability of teacher to teach thesubject



MOOCs

Flexibility to view content anytime, anywhere

Short videos, quizzes, assignments, exams, etc.

On-line forum for peer support and discussion

Advantages

Can be delivered to millions of students via web

A lone professor can support a large class with the help of TAs

Disadvantages


Learning habit of students can be different[1]



MOOCs




Advantages



Disadvantages





MOOCs




Advantages



Disadvantages





Blended MOOCs

Blend of traditional learning and MOOCs

IIT Bombay has started providing these courses throughIITBombayX[2]

Advantages

Includes most of the advantages of traditional learning and MOOCs

Solves many challenges in traditional learning and MOOCs

Various institutes in India are partnering with IIT Bombay to improvelearning of students

Disadvantages

Learning habit of students is not yet shown

Student’s performance analysis is not yet done completely, that isrequired to give feedback/suggestions



Blended MOOCs



Advantages




Disadvantages





Blended MOOCs



Advantages




Disadvantages




Introduction Challenges in blended learning

Challenges in blended learning

Consolidating marks for local and on-line assessment

Mr. Rajeev Gautam (M.Tech student) has been working on this

Providing performance analysis of students

Teachers from participating institutes should get performancesummary of their students

Comparison between on-line and in class learning is required


Introduction What is performance analysis?

What is performance analysis?

Kind of questions that we need to ask:

How many students are solving questions before going through studymaterial?

What are grades of students for a particular class in comparison withother students?

Is there any relation between performance on MOOC and classroomlearning?

Any other feedback that might help for better learning?


Introduction Problem Statement

Problem Statement

Design and implement a system to facilitate performance analysis ofstudents from different participating institutions. Using this system, ateacher from such an institute will be able to

Compare performance of local students with that of other students.

Compare performance of students in local and online assessments.

View the event logs analytic to compare learning habits of students.


IITBombayX Architecture





Open edX is a web-based platform for creating, delivering, andanalyzing online courses

IITBombayX uses architecture of open edX to provide MOOC facility

Blended model is integrated with open edX by providing separateauthentication mechanism


IITBombayX Architecture Various Components

Various Components

CMS (Content management system): This component allows forthe authoring of tools

LMS (Learning Management System): The part of Open EdXthat students interact with, displays content, runs quizzes, andinteractive applications

Event Tracking: Store each occurrence of interaction with systembased on events

Open edX Insights: Separate tool provided by Open EdX forperformance analysis


IITBombayX Architecture Data Modules

Data Modules



Tacking Logs

Every interaction with system is stored as JSON object

Logs can be classified based on event type

These logs are semi structured and very large in size



Event Types

Enrollment Event

Navigational Events

Video Interaction Events

Textbook Interaction Events

Problem Interaction Events

Discussion Forum Events



Sample log record for video interaction event



Common Fields

Context: contains course id, organization id, path, user id, etc.

Event Source: browser or mobile device

Page: URL of the page, the user was visiting when the event wasemitted

Time: UTC time at which the event was emitted

Event Type

User Name, etc.


Proposed Approach and Prototype

Proposed Approach and Prototype


Proposed Approach and Prototype Proposed Approach

Proposed Approach

Important points

Make use of available data to analyze performance of students

Except tracking logs data, other data modules are structured

Open EdX Insights preprocesses and cleans the tracking log data

Only video events are captured as of now by Open EdX Insights

Pre-requirements

To get complete information, pre-process the tracking logs for eachevent type

Clean this information and save for analysis


Proposed Approach and Prototype Prototype

Prototype

Architecture of system proposed


Proposed Approach and Prototype Prototype

Prototype

Objects and tables created

For each event, corresponding Java object, and table are created

Corresponding fields for these objects are also created to storeadditional information

Objects

EventCourseInteract, EventEnrollment, EventForumInteract, UserSession,EventProbInteract, EventVideoInteract, StudentCourseGrade, etc.


Proposed Approach and Prototype Data Analytics

Data Analytics

Data generated from logs in actual implementation will be moved toHDFS.

Using this we will be able to get timeline of a student. This can thenbe used for identifying learning style of a student.[5]

Open edX Insights, with addition of events which were notconsidered, can be used for MOOCs analysis. To use this for blendedmodel, we need to wrap results using authentication mechanismprovided for blended model.


Proposed Approach and Prototype Data Analytics

Typical Analytic Report

Weekly student engagement chart displays the number of students whoengaged in different activities over time in some particular course.

Source: Open edX Insights[7]


Observations

Observations


Observations

Summary of processed logs

Around 20 million IITBombayX tracking log records were processed.

There were JSON objects which were not in proper format arerecorded.

There were some events which are not documented in Open EdXdocumentation.


Observations Invalidated JSON records

Invalidated JSON records

Some log records are not properly structured

Unable to parse these objects and get required data

Issue with code from where these logs are generated


Observations Invalidated JSON records

Sample Invalidated JSON record


Observations Unclassified events/fields

Unclassified events/fields

We found some records in which, their event/fields are not in Open EdXdocumentation.For the purpose of understanding and storing these in tables, these areclassified in existing events based on their nature.

Navigational: goto position, dashboard, jsi18n, i18n.js, jump todiscussion, progress, view courses, logout, how it works, calculate,jump to vertical, etc.

Video interaction events: save user state, transcript translation,transcript download, /transcript/translation, /transcript/download,etc.

Discussion forum events: users, reply, upvote, flagAbuse, follow,unfollow, upload, etc.


Observations Unclassified events/fields

Reporting issues

It is proposed to examine these issues in the context of the new Cypressrelease. Some of the points may have been already addressed. FinalCypress documentation related to Insights is expected to be released soon.Points which are found to be relevant will be addressed in this project, andwill be communicated to Open edX.


Future Work

Future Work


Future Work

Future Work

Tasks:

One pending task in tracking log cleaning is to complete thepreprocessing of discussion forum event

Use the generated data to provide performance reports to teachers.

Finding answers to these questions:

How can we use student timeline to learn about their learning style.

Can we give some suggestions to students for improving theirlearning?


Future Work

References

Thanasis Daradoumis, and Roxana Bassi, and Fatos Xhafa, and Santi Caballe(2013)

A review on massive e-learning (MOOC) design, delivery and assessment

P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), 2013 EighthInternational Conference on 208 – 213

Deepak B Phatak (2015)

Adopting MOOCs for quality engineering education in India

Proceedings of the International Conference on Transformations in EngineeringEducation 208 – 213

Open edX Architecture

https://open.edx.org/contributing-to-edx/architecture

Retrieved on 2015-10-11

Events in the Tracking Logs

http://edx.readthedocs.org/projects/devdata/en/latest/internal data formats/tracking logs.html



Future Work

References

Cristobal Romero, and Sebastian Ventura (2007)

Educational data mining: A survey from 1995 to 2005

Expert systems with applications 33(1), 135 – 146.

Using edX Insights

http://edx-insights.readthedocs.org/en/latest/index.html


Sample image for data analytics report

https://open.edx.org/sites/default/files/wysiwyg/Engagement Chart.png


Edx Analytics Dashboard

https://github.com/edx/edx-analytics-dashboard



Future Work

Thank You...


Documents

Performance Analysis for Blended MOOCs on IITBombayX...Performance Analysis for Blended MOOCs on IITBombayX Rahul Dev Parashar Guided by: Prof. Deepak B Phatak Indian Institute of