Visual Analytics of MOOCs at Maryland Zhengzheng Xu

Computer Science Univ. of Maryland zzxu@cs.umd.edu

Dan Goldwasser CS / UMIACS

Univ. of Maryland goldwas1@umiacs.umd.edu

Benjamin B. Bederson CS / UMIACS

Univ. of Maryland bederson@cs.umd.edu

Jimmy Lin iSchool / UMIACS Univ. of Maryland

jimmylin@umd.edu

ABSTRACT We use visual analytics to explore participation in five MOOCs at the University of Maryland. In some of these courses, our analysis reveals interesting clustering patterns of student behavior. For other courses, visualizations pro-vide “color” to help us better understand the range of stu-dent behavior.

Author Keywords MOOCs; Massive Open Online Courses; Coursera; Learn-ing analytics; Evidence-based education.

ACM Classification Keywords K.3.1. Computer Uses in Education: Distance learning

INTRODUCTION Over 400,000 students have participated in five Coursera Massive Open Online Courses (MOOCs) at the University of Maryland over the past year. Given the widely reported criticism of low success rates of MOOCs in general [3, 5, 8], we wanted to better understand student behavior and its relationship to student success. This short paper visually describes the essential characteristics of student participa-tion in MOOCs at Maryland, along with an analysis of the characteristics of student behavior that correlate with re-ceiving high final grades.

There have to date been a handful of descriptive analyses of MOOC offerings, and some have examined characteristics of students and their activities that correlate with success in the course. Brewslow et al. [2] found very little relationship between student profiles and students’ success. In particu-lar, they found no relationship between age or gender and achievement, and only a “marginal” relationship between highest degree earned and achievement. However, they found that collaborating offline on course material did have a beneficial effect [1]. That study, along with one by Seaton et al. [6] analyzing the same EdX course on “Circuits and Electronics”, examined student activity and reported on the types of activities that students who completed the course engaged in. Instead, we are interested in looking at how student activity correlates with high course grades.

Our interests are more in line with research by Kizilcec et al. [4] that looked at different types of student activity. By analyzing what students do, they identified four prototypi-cal trajectories of engagement, summarized as “auditing” (engaged in the course, but not submitting assessments), “behind”, “on track”, or “out”. This categorization enabled them to look at overall student activity as well as observe changes in behavior.

We explore correlations with course final grades by visually analyzing student participation data to look for trends that might not be obvious based solely on a statistical analysis. Visual analytics is often used to help the investigator decide what questions to ask [1]. When interesting or unexpected patterns emerge, the visual analysis can be coupled with more traditional statistical analyses to provide deeper in-sight into underlying causal mechanisms. For this work, we used Tableau [7], a commercial interactive visualization tool to explore the data, and we report here some of the interesting trends we observed.

DESCRIPTION We analyzed the standard Coursera data available to each course instructor. Our first set of plots was based on data extracted from web activity logs of students. The second plot comes from the higher-level MySQL database provid-ed by Coursera. In both cases, we wrote straightforward scripts to extract and transform the data into a format suita-ble for visualizing.

The most useful representation of a course that we found was a plot of “course success” (as measured by the stu-dent’s final grade) against various types of activity in the course. Here, we show illustrative data from the course ti-tled “Genes and the Human Condition (From Behavior to Biotechnology)”, which had 23,564 registered students in its first offering during April/May, 2013. Using an interac-tive tool such as Tableau enabled us to experiment with many variations of what data to plot and how to visually arrange them. We used “multiple miniatures”, created heat maps, etc. The biggest value was clearly in the ability to interactively explore the data. Below, we show a few key images that provide an indication of what is possible.

The goal of the first exploration was to look at grade by activity. The three images below show grades along the vertical axis and the horizontal axis shows: number of quiz clicks (left), number of lecture page views (middle), and number of forum visits (right).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). L@S 2014, Mar 04-05 2014, Atlanta, GA, USA ACM 978-1-4503-2669-8/14/03. http://dx.doi.org/10.1145/2556325.2567878

L@S 2014 • Poster March 4–5, 2014, Atlanta, Georgia, USA

195

These images show two interesting patterns. The first is that there are three distinct clusters of students – most readily seen in the left plot that shows grade against quiz submis-sions. We hypothesize that these clusters correspond to the “auditing” (at the bottom), “behind” (in the middle) and “on track” (on top) prototypical trajectories identified by Ki-zilcec et al. [4]. This is interesting in that student activities can indeed be categorized into discrete “buckets”.

The second pattern is that the shape of the clusters varies by student activity. For example, the left plot shows that no one receives a good grade without submitting quizzes (which is perhaps obvious). However, from the right plot, we see that students receive good grades without participat-ing in the forums. The middle plot shows that students re-ceive good grades with widely varying numbers of lecture views. These results show that there are multiple ways to be successful in a course, and it is perhaps not necessary to “do everything” – at least, not for everyone.

Student Profiles The next thing we examined was how students’ self-reported “profiles” interacted with performance. As can be seen in the image below, and consistent with Brewslow et al. [2], students of all types appeared in all parts of the plot:

Based on a quantitative analysis of student activities, we did notice some subtle differences in certain types of activity. For example, active forum participants are much more like-ly to be hobbyists and academics than students, industry professionals, or research scientists. And consistent with the literature, forum participants are over twice as likely to be men than women.

CONCLUSION In this short paper, we show how modern visualization tools can provide an effective way to explore large amounts of MOOC data quickly. Interesting trends can then be fol-lowed up with traditional statistical analyses, but we argue that visualizations are a faster and more efficient way of “getting started”.

From our data, we observed three distinct clusters of stu-dents whose behavior may be consistent with other interac-tion styles reported in the literature. However, more work is needed to confirm these initial findings. Next steps include considering how interventions through interface design or explicit “automated encouragements” might increase be-haviors associated with positive outcomes.

ACKNOWLEDGMENTS We appreciate the great efforts of the University of Mary-land instructors for creating these free MOOCs and for the staff on campus for supporting their development. We es-pecially thank Tammatha O’Brien and Raymond St. Leger for sharing their course data with us.

REFERENCES 1. Bederson, B. B., & Shneiderman, B. (Eds.). (2003). The

craft of information visualization: readings and reflec-tions. Morgan Kaufmann.

2. Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., and Seaton, D. T. Studying learning in the worldwide classroom: Research into edX’s first MOOC. Research and Practice in Assessment 8 (Summer 2013).

3. Jordan, K. MOOC Completion Rates: The Data [Web-site]. (Accessed Jan 2, 2014) http://www.katyjordan.com/MOOCproject.html

4. Kizilcec, R. F., Piech, C., and Schneider, E. Decon-structing disengagement: analyzing learner subpopula-tions in massive open online courses. In LAK ’13, ACM (2013), 170–179. http://dl.acm.org/citation.cfm?doid=2460296.2460330

5. Quillen, I. Why Do Students Enroll in (But Don’t Com-plete) MOOC Courses? [Web log post]. (2013). http://blogs.kqed.org/mindshift/2013/04/why-do-students-enroll-in-but-dont-complete-mooc-courses/

6. Seaton, D. T., Bergner, Y., Chuang, I., Mitros, P., and Pritchard, D. E. Who does what in a massive open online course? Communications of the ACM, to appear (2014).

7. Tableau. http://www.tableausoftware.com/ [Accessed January 2, 2014]

8. Westervelt, E. The Online Education Revolution Drifts Off Course. [Radio program]. National Public Radio (December 31, 2013). http://www.npr.org/2013/12/31/258420151/the-online-education-revolution-drifts-off-course

L@S 2014 • Poster March 4–5, 2014, Atlanta, Georgia, USA

196