Towards Strengthening Links between Learning Analytics and Assessment

Dragan Gašević@dgasevic

ETCPS 2017November 16, 2017Iowa City, USA

Feedback loops between students and instructors

are missing/weak!

Learning environment

Educators

LearnersStudent

Information Systems

Blogs

Videos/slides

Mobile

Search

Educators

Learners

Networks

Student Information

Systems

Learning environment

Blogs

Mobile

Search

Networks

Educators

LearnersStudent

Information Systems

Learning environment

Videos/slides

Data in education not new, but…

Real-time insights and longitudinal nature

Open ended and general purpose environments

Different from custom made environments (e.g., ITS)

Assessment and feedback for learning

Student retention

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Year 1 Year 2 Year 3 Year 4

Course Signals

No Course Signals

Arnold, K. E., & Pistilli, M. D. (2012, April). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267-270).

Current state

Understanding and supporting learning

Moving away from deficit models

http://ontasklearning.org

Personalized feedback at scale

Analytics-based feedback

Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology

Analytics-based feedback

Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology

Analytics-based feedback

Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology

High interest in adoption of learning analytics

Is everything all that shiny?

CHALLENGES IN LEARNING ANALYTICS

Challenge

Validity of learning analytics

Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x

Generalizability

Many inconsistent results

Open Academic Analytics Initiative

http://nextgenlearning.org/grantee/marist-college

Consequentiality

Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414-2422.

Can teaching be improved?

Inconsistent associations of network centrality on performance

External validity

Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.

Purposeful measurement

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

How can we act based on the count of logins?

Purposeful measurement

Who makes decisions about instrumentation?

Structural validity

Do existing* measures correspond to trace-based measures?

*mostly self-reported

Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic Performance. Journal of Learning Analytics, 4(2), 113–128.

Understanding learning strategies

Detection of learning tactics Detection of learning strategy

Fincham, E., Gašević, D., Jovanović, Pardo, A. (2017). Seeing the Invisible: Learning Analytics to Measure the Effect of Interventions on Learning Strategies. IEEE Transactions on Learning Technologies (submitted).

Progression

How do we measure progression?

Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., Siemens, G. (2016). Towards Automated Content Analysis of Discussion Transcripts: A Cognitive Presence Case,” Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 15-24).

Cognitive presence

DIRECTIONS

Strengthening links between learning analytics and assessment

Critical dimensions

Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice. Learning: Research and Practice, 3(2), 63-78. doi:10.1080/23735082.2017.1286142

Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology use in predicting academic success. The Internet and Higher Education, 28, 68–84.

Generalizability

Instructional conditions shape learning analytics results

Purposeful measurement

Extraction of theoretically informed traces

Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

Purposeful measurement

Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

Structural validity

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004

Achievement goal

orientation (2x2)

Quid pro quo

Quid pro quo

External validity

Network centrality with weak ties creates advantage only

Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance: importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 314-323). ACM.

Measurement of engagement

Joksimović, S., Poquet, O., Kovanović, V., Dowell, D., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., Brooks , C. (2017). How do we measure learning at scale? A systematic review of the literature. Review of Educational Research (in press).

Tracking progression

Trace data based measures ofthe crowd-sourced learning skill

E.g., Dreyfus model of skill acquisition

Milligan, S. (2015). Crowd-sourced learning in MOOCs: learning analytics meets measurement theory. In Proceedings of the 5th International Conference on Learning Analytics And Knowledge (pp. 151-155). ACM.

Tracking progression

Topic modeling to extract Guttman scales from online discussions

He, J., Rubinstein, B. I., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016). MOOCs Meet Measurement Theory: A Topic-Modelling Approach. Proceedings of the 30th AAAI Conference on Artificial Intelligence (pp. 1195-1201).

FINAL REMARKS

Opportunities afforded by continuous streams of data

Data science methods can be helpful but not sufficient

von Davier, A. A. (2016). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3-11.

Links with methods from assessment and psychometric needed

Tolerance to some measurement imperfections

Generalizability in machine learning is an open research challenge too

Towards Strengthening Links between Learning Analytics and Assessment

Dragan Gašević@dgasevic

ETCPS 2017November 16, 2017Iowa City, USA