Learner Analytics: Hype, Research and Practice in moodle

US West Coast Moodle Moot 2012

John Whitmer, CSU Chico (& Office of the Chancellor)Michael Haskell, Cal Poly San Luis Obispo

Hillary Kaplowitz, CSU Northridge

Learner Analytics: Hype, Research and Practice in Moodle

Download slides at: http://bit.ly/QttGnd

2

“But everything we know about cognition suggests that a small group of people, no matter how intellingent, simply will not be smarter than the larger group. ... Centralization is not the answer. But aggregation is.”

- J. Surowiecki, The Wisdom of Crowds, 2004

Outline

1. Hype & Promise of Learner Analytics

2. Campus Case Studies– Getting Started w/Institutional Reporting (Mike)– Analytics at work in the classroom (Hillary)– Evaluating course redesign (John)

3. Q & A

1. HYPE & PROMISE OF LEARNER ANALYTICS

John Goodlad’s Place-Based Research

Classroom-based research: “What is schooling?”

1,000 classrooms, 27,000 individuals

14 foundations needed to support

Fundamental changes to understanding of educational practice

Steve Lohr, NY Times, August 5, 2009

Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.

8

Source: jisc_infonet @ Flickr.com

Source: jisc_infonet @ Flickr.com

Learner Analytics

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

Fundamental Questions behind Learner Analytics

1. How are students using technology?

2. Does it matter (re: achievement, engagement, learning)?

3. How does this relationship vary (by student, by course, by goal)?

4. What should we do? – Changes in student behavior? – Changes in faculty/program?

L.A. Empirical Research Findings

2. CAMPUS CASE STUDIES

Download slides at: http://bit.ly/QttGnd

Getting Started with Institutional Reporting

Michael Haskell

Cal Poly, San Luis Obispo

We perform Analytics, but are we doing Learning Analytics?

?

What can we do in the meantime…

How far away is Learning Analytics?

Sounds like it’s about 2-3 Years out…

Can we wait that long?

Institutional Reporting

What information is available?

Where is it?

How can we use it?

Types of Information

Individual Behavior

Content Population Behavior

Location of Information

Web Server Logs Population

Moodle Log Table (mdl_log)IndividualContent Populatio

n

Google Analytics Population

Moodle Database Structure Content Individual

http://docs.moodle.org/dev/Database_schema_introduction

Moodle Database Structure

Modules by Course SQL: https://gist.github.com/3203120

How do we utilize this information?Foster collaboration between Faculty

“ Top 10 Instructors TabIn this section, the data was further categorized to find the top 10 instructors in each college who “used the most modules” and “created the most of each module”. The first two graphs show the top 10 instructors from all the colleges.In the first graph, the instructors who used the most modules (8 modules) were X and Y, who are from the College of Engineering and College of Ag, Food and Env respectively. In that same section, Z from the College of Science and Math is listed down three times for classes in the top 10. ”- Student Researcher

How do we utilize this information?To keep a pulse on adoption

How do we utilize this information?

Percentage of Activated Courses by College (Spring 2012)

16%

84%

College of Agriculture, Food and Environmental Science

29%

71%

College of Architecture & Envi-ronmental Design

53%47%

Orfalea College of Business

21%

79%

College of Engineering

26%

74%

College of Liberal Arts

37%

63%

College of Science & Mathematics

To keep a pulse on adoption

How do we utilize this information?To learn how instructors leverage Moodle.

Determine where developer time is best spent.

How do we utilize this information?

Moodle Admin: There’s a problem with Module X.

Instructional Designer: The problem will be fixed soon, but in the meantime I have a workaround I’d like to communicate to instructors. Hmm… I don’t want to reach out to every instructor. Can you provide a list of all the instructors who use Module X?

Moodle AdminNo Problem.

Informed Communication

Conclusions

• Current• Manual Exploration• A lot of Small Wins

• Future• Automate reporting of top tens• Open up the data to a wider audience• Take action on data we have• Keep an eye on LA Tools for faculty and

students

How can data help teachers and students work better

together?

Hillary Kaplowitz

Instructional Designer, Faculty Technology Center

Part-Time Faculty, Cinema and Television Arts Department

California State University, Northridge

Case #1

“I'm not upset that you lied to me, I'm upset that from now on I can't believe you.”

Friedrich Nietzsche

“Hey Professor,

I just looked at my assignments and realized that my Chapter 11 summary did not get submitted, which I'm having trouble believing that I didn't submit it... especially because I see that I did it, and I always submit my assignments as soon as I finish them.”

Now the hard part….

Do I believe him?

If I only I could check…

And it was all his idea…

The student suggested that I check Moodle and if that didn’t work told me how to check the Revision History in GoogleDocs with step-by-step directions!

Case #2

“Life isn't fair. It's just fairer than death, that's all.”

William Golding

“The quiz is unfair”

Hybrid Course Weekly Structure

1. Watch lectures

2. Read textbook

3. Online chat and tutoring

4. Post questions and take practice

quiz

5. Class meets

6. Aplia quiz

But the story was not that simple…

• Reports on Moodle painted a different picture• Student was watching the lectures at 10:00 p.m.• Then immediately taking quiz

Enabled constructive feedback…

Advised the student how the structure of the course was designed to enhance learning

Student revised their study habits Improved grades and thanked the instructor!

What we can do with data now

Use Reports in Moodle to verify student claims Review participant list to see last access time Empower students to review their own reports Analyze usage and advise students how to study better Review quiz results to find common misconceptions

Could we help improve student learning outcomes if we knew the effect of…

Coffee

Sequencing

Amount

Textbook

LMS Access

LMS Activities

Mobile

Attendance

Facebook

43

EVALUATING COURSE REDESIGN: INTRO TO RELIGIOUS STUDIES 180

44

Front-end: What? Why? Evaluation for Program Assessment• Year-long faculty course redesign program

• Case: Intro to Religious Studies: increased enrollment from 80 to 373 students first semester: 250,000 course website hits

• Outcome: increased mastery course concepts AND increased number D/W/F students

• Why? (and for whom?)

• What is the relationship between LMS actions, student background characteristics and student academic achievement? (6 million dollar question)

45

Back-end: How?

• Integrated data from LMS log files, student enrollment records, and course grade

• LMS logfiles are “data exhaust” for server analysis

• Filtering and cleaning reduced 250K records to 71k

• Analysis tools: Excel, Tableau (visualization), Stata (statistical analysis)

46

grade

A A- B+ B B- C+ C C- D+ D F W

0K

5K

10K

15K

20K

25K

30K

Avg. total_dwell0

20

40

60

80

100

120

140

160

180

200

220

240

260

280

Avg. total_hits

Measure Names

Avg. total_dwell

Avg. total_hits

47

Grades by Hits & Dwell Timegrade

A A- B+ B B- C+ C C- D+ D F W

0K

5K

10K

15K

20K

25K

30K

Avg. total_dwell

0

20

40

60

80

100

120

140

160

180

200

220

240

260

Avg. total_hits

Measure Names

Avg. total_dwell

Avg. total_hits

Pell v. Non-Pell: Grades by Hit/Dwellgrade

A A- B+ B B- C+ C C- D+ D F W

0K

10K

20K

30K

Avg. total_dwell

0

50

100

150

200

250

300

Avg. total_hits

Pell Eligible

Pell-Eligible

Not Pell-Eligible

49

Content: the Time Differentialgrade / pelleligible

A B+ C C-

Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible

0K

5K

10K

15K

20K

25K

30K

35K

Value

Content

Content

Engage

Engage

Assess

Assess

Admin

Admin

Content

Content

Engage

Engage

Assess

Assess

Admin

Content

Content

Engage

Engage

Assess

Assess

Content

Content Engage

Engage

Assess

Assess

Admin

Admin

Measure Names

Admin

Assess

Engage

Content

Call to Action

1. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...

2. Metrics reporting is the foundation for Analytics

3. Don’t need to wait for student characteristics and detailed database information; LMS data can provide significant insights

4. If there’s any ed tech software folks in the audience, please help us with better reporting!

http://1.usa.gov/GDFpnI

Draft DOE Report released April 12

52

Q&A and Contact Info

Download slides at: http://bit.ly/QttGndResources Googledoc: http://bit.ly/HrG6Dm

Contact Info: • John Whitmer (jwhitmer@csuchico.edu)• Michael Haskell (mhaskell@calpoly.edu)• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu)

53

Works CitedAdams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning through Educational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department of Education, Office of Educational Technology.Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1). Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved from http://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era. EDUCAUSE Review, 42(4), 17. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path to value. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project: IBM Institute for Business Value and MIT Sloan Management Review.Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data: The next frontier for innovation, competition, and productivity.Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed. Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-debating-the-coming-personalization-of-higher-ed/36057Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved from http://www.learninganalytics.net/