Pipeline for Expediting Learning Analytics and Student Support from Data in Social Learning Yohan Jo, Gaurav Tomar, Oliver Ferschke, Carolyn P. Rosé, Dragan Gasevic {yohanj, gtomar, ferschke, cprose}@cs.cmu.edu, dgasevic@acm.org

C A R N E G I E M E L L O N U N I V E R S I T Y M E R C H A N D I S E B R A N D G U I D E L I N E S | 1/28/2016 3

O F F I C I A L U N I V E R S I T Y S E A L

The official seal of Carnegie Mellon University from 1967 was originally reserved for only official documents, including diplomas, presidential and trustee minutes or other legal, academic or official university documentation — or on the highest awards or certificates.

The seal can now also be used for FORMAL occasions and formal products, including items for commencement, specific gift items in brass, silver or pewter, appropriate clothing (sports coats), stationery and other items. DO NOT use the official seal in combination with the wordmark. If an item is in question, please contact Marketing Communications at marketing-info@andrew.cmu.edu or the Trademark Licensing Office. Please contact Creative Services at commdesign-photog@andrew.cmu.edu to obtain the official seal graphic.

The Official Seal: 4-color, 1-color

4-color version

1-color version less than 2 inchesIf the one color seal (shown here in pms 187) is used at a diameter less than two inches, Carnegie Mellon University appears in a lighter font.

One-color version can be printed in black, PMS 187-red, Metallic PMS 877-silver and 873-gold.

Ideal for embossing and foil stamping.

White version reversed out of a background When reversing the seal out of a color or black background, do not use the 1-color (black) version; always use the reversed (white) version.

Only reverse out on black, PMS 187-red or gray.

1-color version greater than 2 inches If the one color seal is used two inches or greater in diameter, Carnegie Mellon University appears in a bolder font.

10 Chapter 1: Our logo www.ed.ac.uk/brand

The University of Edinburgh corporate logosare also available in black, white and corporate blue (Pantone 282)

Black (horizontal logo)

White version, please note:

The book in the centre of the roundel should be white in the white versions of the logo.

This is different to how it appears in the black version where the book is transparent.

For an example of incorrect usage, please refer to Incorrect use of University of Edinburgh corporate and secondary logos on page 34.

White (horizontal logo)

Corporate blue (horizontal logo)

Motivation Course Context

Data Infrastructureunifies social interactioninto a uniform interface

Learning Process Analysis models learner behaviorsconditioned on social connection

Interventionhelps studentsengage in beneficial social interaction

• The goal is to tighten the analytics cycle of data leading to insights on student needs and improvements in student support.

• We focus on social learning, where students learn through social interaction, e.g., via observation, help exchange, and discussion.

Period: Oct-Dec, 2014Number of Students• edX: 23,000• ProSolo: 1,700

Conventional xMOOC Platform Self-Regulated Learning Platform

Contributions

References

• Detailed Article: Y. Jo, G. Tomar, O. Ferschke, C. P. Rose, D. Gasevic. Expediting Support for Social Learning with Behavior Modeling, EDM ‘16.

• Learning Process Model: Y. Jo and C. P. Rose. Time Series Analysis of Nursing Notes for Mortality Prediction via a State Transition Topic Model. CIKM ’15.

• Recommendation Model: D. Yang, D. Adamson, C. P. Rose. Question recommendation with constraints for massive open online courses. RecSys ’14.

• Course: C. P. Rose, O. Ferschke, G. Tomar, D. Yang, I. Howley, V. Aleven, G. Siemens, M. Crosslin, D. Gasevic, and R. Baker. Challenges and Opportunities of Dual-Layer MOOCs: Reflections from an edX Deployment Study. CSCL ’15.

• More Research & Resources: http://dance.cs.cmu.edu

Findings from Application

• Learning process analysis finds that students who follow goal-setting peers show positive learning behaviors:

• Stay long in the course.• Engage in hands-on practices.• Revisit learning materials across the course.

• Recommender system finds that • Explicit intervention is necessary for helping students be aware of

qualified students and interact with them.• Our algorithm effectively matches qualified students to relevant

discussions while satisfying the constraints.• DiscourseDB eases similar analysis and intervention on different data.

DiscourseDB (http://discoursedb.github.io)• Maps diverse forms of textual conversations and social interactions

into a common structure.• Enables the subsequent components—learning process analysis and

intervention—to apply the same tools to different data with little modification.

Discoursee.g.,forums,socialmedia

Contributione.g.,posts,comments,utterances

Usere.g.,students,usersreply-to*

*DiscourseDB allows defining arbitrary relations between contributions, avoiding data-specific tables.

composes

createsrevisesdeletesfollows

follows

Entity-Relation Model

…

Temporal Bayesian Network• Represents the building blocks (states) of

learning process as• Distribution over discussion topics ( )• Distribution over discussion media ( )• Transition probabilities to other states

for each social connection type ( )

• Propose a pipeline and component models for data infrastructure, learning process analysis, and intervention.

• Demonstrate an application of the pipeline to real data to examine goal-setting behavior as qualifications of role models.

⇡

✓

S! Z!

M!

S!

…"

…"

…"

↵

✓

�

z w

�

…"

π"⇡

…"

…"

d

D!

z w

d

z w

d

…"

�

s1

st�1

st

…"

…"

⌫

Nmt

DiscussionMedia

States TopicsSocial Connection Types

Students

Social Connection Types (e.g.)• Follows goal-setting peers• Follows no one

Discussion Media (e.g.)• Blog, Twitter, Forum

Input• Each student’s discussions and social

connection types over time

Output• For each state: discussion topics and

media, state transition probabilities• For each student: state sequence

Recommender System• Draws upon insights from the learning

process analysis and aims to foster beneficial social connection among students.

• Recommends discussions to qualified students.

• Allows the discussants to interact with the qualified students.

Relevance Prediction• Relevance between students and

discussions is calculated using:‣ Students’ expertise & motivation.‣ Discussion’s length & popularity.

Constraint Filtering• Helpful recommendation also

requires constraints:‣ Every discussion has at least one

qualified student.‣ A student is not overloaded.

Our recommendation system is aimed at recommending dis-cussions to a “qualified” student so that the discussants canhave social interaction with and benefit from this student.The recommendation system has two steps: relevance pre-diction and constraint filtering. The relevance predictionstep learns the relevance between discussions and studentsvia a feature-aware matrix factorization, which predicts therelevance using information, such as student- and discussion-related features, potentially valuable in making recommen-dations. The constraint filtering step combines the learnedrelevance with given constraints (e.g., qualification) to makefinal recommendations via a max concave cost flow network.By performing these two steps, we explore where in the pro-cess these additional features contribute most positively tothe estimated benefit of the recommendation, whether atthe relevance prediction stage or at the constraint filteringstage.

3.3.1 Context-Aware Relevance Prediction

Our algorithm extends the earlier model proposed by Yanget al. [27], to see the impact of some additional statisticalfeatures related to individual students, such as their quali-fication and how well they are connected with their peers.The predicted relevance scores are the basis of the subse-quent constraint filtering phase.

The relevance matrix between students and discussions is de-noted as R with entry ru,d representing the relevance scorebetween student u and activity d. ru,d is 1 if and only if stu-dent u has participated in discussion d. As in the originalmodel, our model mainly exploits three aspects of contex-tual information: student features, discussion features, andimplicit feedback. In addition to them, we add the followingadditional student features.

• Goal quality (�): a student’s goal quality as ex-plained in Section 4.2.

• Centrality ( ): how well a student is connected inthe social network of all students.

Taking these additional features into account, our new context-aware relevance prediction model can be formulated:

ru,d = bias+ (Pu + �u�+ ✓u⇥+ �u⇤+ u + ��)T

(Qd + �d�+ ldL+1p

|U (d) |

X

v2U(d)

'v). (1)

Here, ru,d is the predicted relevance score of u on d. �u and✓u are the number of discussions u has participated in andthe number of discussions u has initiated. �� is a one-hotvector indicating the course week that u registered for thecourse; this may be related to the student’s motivation. �dand ld are the number of replies and the length of the contentin d. U(d) denotes the set of students participating in d,and 'v is the predicted preference v of u. Pu and Qd are thebiases of u and d. �, ⇥, ⇤, , �, and L are one-dimensionalfeature weights. The inference step of the parameters areavailable on our website, and interested readers are referredto the original paper [27].

3.3.2 Max Cost Flow Constraint Filtering

It is important to recommend relevant discussions to a stu-dent, but it would be better if the student is beneficial tothe discussants and thereby optimize the overall communitywelfare. For this purpose, we use a max cost flow model tosubject the relevance scores obtained in the previous stepwith the following constraints on students and discussions.Here fu,d 2 0, 1 is an indicator of whether discussion d isrecommended to student u.

1. Goal quality: For every discussion d, at least onestudent u to which we recommend d should have agoal quality Gu greater than some threshold G. Inaddition, the larger Gu the better. We will see why agoal quality can be a qualification in Section 4.3.

2. Centrality: For every discussion d, at least one stu-dent u to which we recommend d should have a cen-trality score Su greater than some threshold S. In ad-dition, the larger Su the better. This constraint is forhelping the discussants make more social connectionthrough the student brought to the discussion.

These constraints are formulated into the following opti-mization problem.

maxX

u,d

fu,d · ru,d � ↵ ·X

d

X

u

(Gu · fu,d � G)(Gu �G)

�↵ ·X

d

X

u

(Su · fu,d � S)(Su � S) s.t.

8d 2 D, 9u 2 U,Gu · fu,d � G

8d 2 D, 9u 2 U, Su · fu,d � S (2)

D and U are the sets of discussions and students, respec-tively. is an indicator function. ↵ balances the importanceof the constraints. To maximize this objective function, weconstructed a concave cost network. The details are avail-able on our website, and interested readers are referred tothe original paper [27].

4. CASE STUDYIn this section, we describe our case study, where we appliedour pipeline to social learning data and investigated the waysocial connections a↵ect students’ learning paths and howthe insights can inform the recommendation of discussionthreads in forums. We start by describing the course contextand the categories of social connections.

4.1 Course Context and Data MappingThe data used in this paper was collected in the contextof an edX MOOC entitled Data, Analytics, and Learning(DALMOOC) [17], which ran from October to December2014. This MOOC was termed a dual layer MOOC be-cause students had the option of choosing a more standardpath through the course within the edX platform or to fol-low a more self-regulated and social path in an external en-vironment called ProSolo. While the edX layer was morescripted and focused on the traditional video and assign-ment paradigm, the ProSolo layer allowed students to settheir own learning goals and follow other students so thatthey can have easy access to their followees’ activities anddocuments. This course covered theoretical principals aboutlearning analytics as well as tutorials on social network anal-ysis, text mining, and data visualization.

Student features

Discussion features

Relevance

Penalty on workload

Our recommendation system is aimed at recommending dis-cussions to a “qualified” student so that the discussants canhave social interaction with and benefit from this student.The recommendation system has two steps: relevance pre-diction and constraint filtering. The relevance predictionstep learns the relevance between discussions and studentsvia a feature-aware matrix factorization, which predicts therelevance using information, such as student- and discussion-related features, potentially valuable in making recommen-dations. The constraint filtering step combines the learnedrelevance with given constraints (e.g., qualification) to makefinal recommendations via a max concave cost flow network.By performing these two steps, we explore where in the pro-cess these additional features contribute most positively tothe estimated benefit of the recommendation, whether atthe relevance prediction stage or at the constraint filteringstage.

3.3.1 Context-Aware Relevance Prediction

Our algorithm extends the earlier model proposed by Yanget al. [27], to see the impact of some additional statisticalfeatures related to individual students, such as their quali-fication and how well they are connected with their peers.The predicted relevance scores are the basis of the subse-quent constraint filtering phase.

The relevance matrix between students and discussions is de-noted as R with entry ru,d representing the relevance scorebetween student u and activity d. ru,d is 1 if and only if stu-dent u has participated in discussion d. As in the originalmodel, our model mainly exploits three aspects of contex-tual information: student features, discussion features, andimplicit feedback. In addition to them, we add the followingadditional student features.

• Goal quality (�): a student’s goal quality as ex-plained in Section 4.2.

• Centrality ( ): how well a student is connected inthe social network of all students.

Taking these additional features into account, our new context-aware relevance prediction model can be formulated:

ru,d = bias+ (Pu + �u�+ ✓u⇥+ �u⇤+ u + ��)T

(Qd + �d�+ ldL+1p

|U (d) |

X

v2U(d)

'v). (1)

Here, ru,d is the predicted relevance score of u on d. �u and✓u are the number of discussions u has participated in andthe number of discussions u has initiated. �� is a one-hotvector indicating the course week that u registered for thecourse; this may be related to the student’s motivation. �dand ld are the number of replies and the length of the contentin d. U(d) denotes the set of students participating in d,and 'v is the predicted preference v of u. Pu and Qd are thebiases of u and d. �, ⇥, ⇤, , �, and L are one-dimensionalfeature weights. The inference step of the parameters areavailable on our website, and interested readers are referredto the original paper [27].

3.3.2 Max Cost Flow Constraint Filtering

It is important to recommend relevant discussions to a stu-dent, but it would be better if the student is beneficial tothe discussants and thereby optimize the overall communitywelfare. For this purpose, we use a max cost flow model tosubject the relevance scores obtained in the previous stepwith the following constraints on students and discussions.Here fu,d 2 0, 1 is an indicator of whether discussion d isrecommended to student u.

1. Goal quality: For every discussion d, at least onestudent u to which we recommend d should have agoal quality Gu greater than some threshold G. Inaddition, the larger Gu the better. We will see why agoal quality can be a qualification in Section 4.3.

2. Centrality: For every discussion d, at least one stu-dent u to which we recommend d should have a cen-trality score Cu greater than some threshold C. Inaddition, the larger Cu the better. This constraint isfor helping the discussants make more social connec-tion through the student brought to the discussion.

These constraints are formulated into the following opti-mization problem.

maxX

u,d

fu,d · ru,d � ↵ ·X

d

X

u

(Gu · fu,d � G)(Gu �G)

�↵ ·X

d

X

u

(Cu · fu,d � C)(Cu � C) s.t.

8d 2 D, 9u 2 U,Gu · fu,d � G

8d 2 D, 9u 2 U,Cu · fu,d � C (2)

D and U are the sets of discussions and students, respec-tively. is an indicator function. ↵ balances the importanceof the constraints. To maximize this objective function, weconstructed a concave cost network. The details are avail-able on our website, and interested readers are referred tothe original paper [27].

4. CASE STUDYIn this section, we describe our case study, where we appliedour pipeline to social learning data and investigated the waysocial connections a↵ect students’ learning paths and howthe insights can inform the recommendation of discussionthreads in forums. We start by describing the course contextand the categories of social connections.

4.1 Course Context and Data MappingThe data used in this paper was collected in the contextof an edX MOOC entitled Data, Analytics, and Learning(DALMOOC) [17], which ran from October to December2014. This MOOC was termed a dual layer MOOC be-cause students had the option of choosing a more standardpath through the course within the edX platform or to fol-low a more self-regulated and social path in an external en-vironment called ProSolo. While the edX layer was morescripted and focused on the traditional video and assign-ment paradigm, the ProSolo layer allowed students to settheir own learning goals and follow other students so thatthey can have easy access to their followees’ activities anddocuments. This course covered theoretical principals aboutlearning analytics as well as tutorials on social network anal-ysis, text mining, and data visualization.

Every discussion needsa qualified student

Relevance

DiscussionAffordances for Natural Collaborative Exchange

• Allows annotating entities and text spans manually or automatically.

• Keeps track of changes in relationships between entities and in the content of textual contributions.