The indicators project - ASCILITE version

The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors

Colin BeerDavid JonesKen Clark

http://tr.im/FvRQ

Twitter: #eair

http://www.flickr.com/photos/anders-vindegg/3408838186/

Surprising patterns in CQU LMS data

Apparent benefit in LMS usage analysis that is

Different from established patterns

Due to limitations in current work?

Cross-LMS

Cross-institutional

Longitudinal

Want to play?

Overview

http://www.flickr.com/photos/atbaker/1577665185/

Context

Level of feature adoption

Link between participation & grades

Reflection and Future work

Overview


Context




http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

2009)

http://www.flickr.com/photos/mikebaird/2087879492/

(Dawson, 2009; Dawson et al, 2009)

(Bakharia et al, 2009)

(Malikowski, 2008; Malikowski, et al, 2007;Malikowski et al, 2006)

(Campbell et al, 2007)

(Griffiths, 2007)

(Dawson, 2006)

(Black et al, 2008)

(Dawson et al, 2008)

(Heathcoate et al, 2005)

(Hornik et al, 2008)

(Romero et al, 2007)

http://www.flickr.com/photos/mwichary/2188958154/


LMS reporting toolsOnly what is in the

LMS

Limited views


Some surveysHuman

recollectionLMS reporting toolsOnly what is in the

LMS

Limited views


Some surveysHuman

recollection

Manual checksTime-

consuming

Error prone


LMS

Limited views


Some surveysHuman

recollection

Limited longitudinal data

Manual checksTime-

consuming

Error prone


LMS

Limited views


Some surveysHuman

recollection


Limited cross-LMS comparisons

Manual checksTime-

consuming

Error prone


LMS

Limited views


Some surveysHuman

recollection


Limited cross-LMS comparisons

Limited cross-institutional comparisons

Manual checksTime-

consuming

Error prone


LMS

Limited views

“LMS” situation

Students

Webfuse 1997-2009

Pre-2010 “LMS”

WebCT 1999-2004

Blackboard 2004-2009

Webfuse 1997-2009

Post-2010 “LMS”

Blackboard 2004-2009

Moodle….

Students - 2009

CQ on-campus

4436

Students - 2009

CQ on-campus

Distance education

4436

8444

Students - 2009

CQ on-campus

Distance education

AIC on-campus

4436

8444 796

2

http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

2009)

http://indicatorsproject.wordpress.com

Enabling comparisons of LMS usageacross institutions, platforms and

time

http://www.flickr.com/photos/david_jones/36379215/

Overview


Context




http://www.flickr.com/photos/chokingsun/3495110670/

There are more similarities than differences among LMS (Black et al,

2007)


TransmittingContent

CreatingClass

Interactions

ComputerBased

Instruction

Move to features in unexplored categories, until each category is considered for different learning needs

Level 1: Most used Level 2: Moderately used Level 3: Rarely used

• Content/files• Announcements• Gradebook

More than 50% of courses More than 20% of courses (much) less than 20% of courses

EvaluatingStudents• Quizzes• Drop box

Evaluatingcourse and instructors

• Surveys• Chat• Discussion forum• Email

• Some quizzes• Adaptive release

(Malikowski et al, 2007)


What we did

CQU

Wf BbLMS logs


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


% of courses adopting “Malikowski features”


Year Blackboard Webfuse2005 777 3042006 810 2962007 874 2522008 945 2252009 868 174

# course sites per “LMS”


Malikowskitop & bottomrange


Blackboard


Webfuse




http://www.flickr.com/photos/wadem/2730257498/


The particular trajectory of emergence is not wholly

determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two(Jones,

1999)

http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

Will Moodle be different?

Why?

What about other institutions?

Are Malikowski ranges out of date?

When is a feature adopted?

How

…….

Overview


Context




http://www.flickr.com/photos/wolfgangstaudt/2279651479/

The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course (Dawson et al,

2008)

Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance. (Dawson, 2009; Morris et al,

2005)

http://www.flickr.com/photos/jamuraa/813966437/

What about

Different student types?

Different levels of staff interaction?Different staff academic background?

(Fresen, 2007)

CQU context


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage

Student records


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage

Participation, grades,external factors

Student records

Participation

Hits

Discussion forum

Entire course siteJust course forum

# replies# posts

http://www.flickr.com/photos/adactio/1259245482/

N = 510,158

Student/courses



Mixed design ANOVA

All reported effects are significant

Significant interaction effectbetween grades and participation


Hits on course site

Hits on course forum

Distance education students


CQ on-campus students


AIC students


AIC

CQ

FLEX


Distance education students

# of replies

# of posts


CQ on-campus students


AIC students



2008)


2005)



What about



(Fresen, 2007)

CQU context

http://www.flickr.com/photos/orinrobertjohn/3321067724/

Course groups based on staff hits

Group Courses Student-courses

Students/course

Definition

High 678(17%)

156,770 231 x>3000 hits

Medium 980(25%)

143,314 146 3000>x>1000

Low 1474(37%)

102,214 69 1000>x>100

Super low 849(21%)

68,354 80 100>x>0


Hits

High staff participation courses


Replies & posts

High staff participation courses


Hits

Super low staff participation courses


Replies & posts

Super low staff participation courses


High

Posts comparisons

Superlow



2008)


2005)


http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

- Course size

Examination of super-low

Networks of interaction (Bakharia et al, 2009; Dawson,

2009)

Forum content


What about



(Fresen, 2007)

CQU context


Course offerings with

Staff with grad certs in L&TSignificant input from instructional designerStaff who have received teaching awards

Staff teaching education coursesNone of the above

n=167

n=4

n=196

n=3288

N=362


None of the above


Grad cert in L&T


Teaching awards


Education discipline


Instructional design

Overview


Context



Refleection and Future work

http://www.flickr.com/photos/mikebaird/2985066755/

Some interesting patterns

Useful idea

Needs more work


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data

Useful Data/Informat

ion


ion


ion


ion

T&L Practice Research


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What


Avg course hits

DE under grad students

By age


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What

Why


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What

WhyHow


http://www.flickr.com/photos/dmatos/540688952/


http://www.flickr.com/photos/sylvar/1138341328/

http://www.flickr.com/photos/fncll/145149313/


http://tr.im/FvRQ

http://indicatorsproject.wordpress.com

Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.

Black, E. W., K. Dawson, et al. (2008). "Data for free: Using LMS activity logs to measure community in online courses." Internet and Higher Education 11(11): 65-70.

Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends 51(2): 35-39.

Campbell, J., P. DeBlois, et al. (2007). "Academic analytics: A new tool for a new era." EDCAUSE Review 42(4): 40-42.

Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.

Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.

Dawson, S., L. Macfadyen, et al. (2009). Learning or performance: Predicting drivers of student motivation. Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.

Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230

Dawson, S. (2006). "Online forum discussion interactions as an indicator of student community." Australian Journal of Educational Technology 22(4): 495-510.

Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning 6(3): 351-362.

Griffiths, M. E. (2007). Patterns of user acitivity in the Blackboard course management system across all courses in the 2004-2005 academic year at Brigham Young University. Department of Instructional Psychology and Technology., Brigham Young. Master of Science: 117.

Heathcoate, L. and S. Dawson (2005). "Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS." E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.

Hornik, S., C. S. Saunders, et al. (2008). "The Impact of Paradigm Development and Course Level on Performance in Technology-Mediated Learning Environments." Informing Science. The international journal of an emerging transdiscipline 11(11).

Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research 36(2): 149-173.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research 36(2): 149-173.

Malikowski, S., M. Thompson, et al. (2006). "External factors associated with adopting a CMS in resident college courses." Internet and Higher Education 9(3): 163-174.

Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education 8(3): 221-231.

Romero, C. b., S. n. Ventura, et al. (2007). "Data mining in course management systems: Moodle case study and tutorial." ScienceDirect Computers and Education (51 (2008)): 368-384.

Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology 13(3): 201-218.

Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.