Should Your MOOC Forum Use a Reputation System? DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY SHOULD

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 1/46

Should Your MOOC Forum Use a Reputation System?DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN

UNIVERSITY OF CALIFORNIA, BERKELEY


One-slide summary• Goal: Improve learning in MOOCs by improving forums


One-slide summary• Goal: Improve learning in MOOCs by improving forums• Reputation systems give points to users who post the most useful questions and responses


One-slide summary• Goal: Improve learning in MOOCs by improving forums• Reputation systems give points to users who post the most useful questions and responses• This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without


One-slide summary• Goal: Improve learning in MOOCs by improving forums• Reputation systems give points to users who post the most useful questions and responses• This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without• Results: Posts in the forum with reputation system received more responses and quicker responses, but led to no significant difference in grades/attrition


Outline• Motivation and Background•MOOC Forums, Reputation Systems



• Experimental Setup•Randomized controlled study, MOOC integration




• Results




• Results• Discussion / Follow-up work


MOOC Forums• Dominant form of student interaction and support in MOOCs today• “an essential ingredient of

an effective online course” (Mak et al 2010)


MOOC Forums• Dominant form of student interaction and support in MOOCs today• Particularly in technical classes, used primarily for answering questions•Administrative, particular

quiz problems, concepts etc.


Reputation Systems• Used by successful Q&A forums on the web•StackOverflow•Yahoo! Answers

• Fast high-quality answers (Mamykina et al, CHI 2011)


Reputation Systems• Used by successful Q&A forums on the web•StackOverflow•Yahoo! Answers

• Fast high-quality answers (Mamykina et al, CHI 2011)• Users have a public numeric reputation• Users receive points when others upvote their questions and/or responses




Reputation Systems: Gaining Privileges• As users gain points, they acquire more moderator privileges on the site (e.g. edit question, retag question, delete question…)


Experimental Setup• Deployed in a single software engineering MOOC on edX• Randomized controlled trial (aka A/B testing)


Experimental Setup• Deployed in a single software engineering MOOC on edX• Randomized controlled trial (aka A/B testing)•Two independent forums•Same software, different reputation features enabled




Experimental Setup: edX Integration

• External website linked from edX• Authenticated using edX OpenID


Statistics and Data Overview• Course had 5985 enrolled and 532 passing students• 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without)


Statistics and Data Overview• Course had 5985 enrolled and 532 passing students• 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without)

Forum withreputation

Forum withoutreputation

Total posts 819 587

Questions (threads)

115 144


Results: Improved forum metrics• Faster response times•Mean of 59 min vs. 2hr20min•Mann-Whitney U (p < 0.001)


Results: Improved forum metrics

• Faster response times•Mean of 59 min vs. 2hr20min•Mann-Whitney U (p < 0.001)

• More responses per post•Mean of 3.5 vs. 2.4

responses•Mann-Whitney U (p < 0.001)


Results: No significant difference in student outcomes

• Grades



• Grades• Retention/Attrition



• Grades• Retention/Attrition• Rovai’s Sense of Community• 20-question survey measuring

how much student feels like“I belong to a community that I can trust and depend on”


Results: Reputation System Decreases Appeals for Help

• Example of analysis based on post content• Examples: “Has anyone else experienced something similar?”,“Any help would be greatly appreciated.”• Theory: no need to appeal for help if system rewards helpers• Looked for posts containing appeals for help•Three raters rated posts as appeal for help or not (κ = 0.80)

• More appeals for help in forum without reputation system•7.3% of posts vs. 2.9% (Fisher’s exact test, p ≈ 0.0002)


Do discussion forums affect learning at all?

• Cheng et al, 2011:•"an online discussion forum was implemented at the beginning of an undergraduate introductory psychology course [...] students who participated in the forum tended to have better performance in the course, and […] participating in the discussion forum, particularly reading posts on the forum, slightly improved exam performance“

• Replicate in MOOC setting?


Subjects – Controlled study

Others(4884)

Forumwith

reputationsystem(548)

Forumwithout

reputationsystem(553)


Subjects – Descriptive/Retrospective study

Others(1192)

Visitedany forum

at least once

(1572)

Viewed at least one lecture, HW, or quiz

(4793)Non-forum

users(3221)

Self-selected


Results: Forum use predicts better student outcomes

• Grades Assignment Forum users have higher

scores?

Significance

Homework 1 Yes p < 0.001Homework 2 Yes p < 0.02Homework 3 Not shown p > 0.1Homework 4 Not shown p > 0.2Quiz 1 Yes p < 0.001Quiz 2 Yes p < 0.001Quiz 3 Yes p < 0.008


• Grades• Retention•Median 39 of 42 days, vs.

19 of 42

Results: Forum use predicts better student outcomes


Discussion• Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes


Discussion• Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes• Possible explanation for lack of significant difference: low participation


Discussion: Low participation• In any given week, 13-17% of students accessing course website accessed experimental forum website


Discussion: Low participation• In any given week, 13-17% of students accessing course website accessed experimental forum website• About half of subjects accessed 5 or less threads ever

• Significant differences in outcomes unlikely when most subjects don’t even use the feature• Also found low participation in work with chats [L@S 2014]

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM?

Limitations

39/46

• Single course offering (generalizable?)

• Small number of very active users (see Teo & Johri, this conference)

• Bias due to consent procedure

• Contamination (outside interaction)


Lessons for field study researchers in MOOCs

• Study retention and performance separately• Student who drops after getting 100% on one quiz may have same final

grade as a student who gets 25% on every quiz





• Don’t rely on late surveys – only hits biased survivor group





• Don’t rely on late surveys – only hits biased survivor group• Use established platforms, working around extensibility limitations


Follow-up work• “Chatrooms in MOOCs: All Talk and No Action” (Learning @ Scale 2014)•Another controlled experiment•Deployed a global synchronous chatroom on every page• Low participation; no significant effect on student outcomes


Follow-up work• “Chatrooms in MOOCs: All Talk and No Action” (Learning @ Scale 2014)•Another controlled experiment•Deployed a global synchronous chatroom on every page• Low participation; no significant effect on student outcomes

• Peer learning chat (in progress)• Students discuss questions in synchronous chat in small groups• Early work with Turk simulations shows users enjoy using it• Planned to be deployed in a MOOC


Questions?

Derrick Coetzee ([email protected])Armando Fox ([email protected])Marti A. Hearst ([email protected])Björn Hartmann ([email protected])

Supported by National Science Foundation under award IIS-1149799 and partially funded by the National Endowment for the Humanities under grant HK-50011

Documents

Should Your MOOC Forum Use a Reputation System? DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY SHOULD