20
W I S S E N T E C H N I K L E I D E N S C H A F T http://elearning.tugraz.at Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Educational Technology & Institute of Theoretical Computer Science Graz University of Technology Edinburgh, 29.04.2016

Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

Embed Size (px)

Citation preview

Page 1: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

W I S S E N T E C H N I K L E I D E N S C H A F T

http://elearning.tugraz.at

Bayesian Modelling of Student Misconceptionsin the one-digit Multiplicationwith Probabilistic ProgrammingBehnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner,Educational Technology & Institute of Theoretical Computer ScienceGraz University of Technology

Edinburgh, 29.04.2016

Page 2: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Presentation Outline

1. Main Idea

2. Previous Work

3. Model Structure

4. Learning the Model’s Parameters

5. Future Work

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20162

Page 3: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Main Idea

Model simple learning misconceptions

Probabilistic graphical models (bayesian networks)

Prediction of future student behaviour

How can this influence learning decision processes?

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20163

Page 4: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Previous Work (1/2)

Derive most prevalent error types in one-digitmultiplication problems

Explore the misconceptions and most difficult vs.easy problems

Cluster one-digit multiplication problems according todifficulty probabilities

Problems: No model, no adaptation, snapshot,inflexible

Solution: Probabilistic Graphical Models

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20164

Page 5: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Previous Work (2/2)

1x1 Trainer: http://schule.learninglab.tugraz.at/einmaleins/ (last access 29 April 2016)

Developed by Graz University of Technology

Applied in different schools in Austria, Germanyand Switzerland

Limited information provided only by the answers,no demographic values

Train the model (Model “learns” its parameters)

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20165

Page 6: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

1x1 Trainer Application

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20166

Page 7: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Difficult vs. Easy Questions

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20167

Page 8: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Average Time Consumption

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20168

Page 9: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Clustering One-Digit Multiplication Problems

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.20169

Page 10: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Derived Error Types

Error type Description e.g. 56 = 7 ∗ 8operand errors a neighbouring operand is takensplit 1 the neighbouring distance is 1 48 = 6 ∗ 8split 2 the neighbouring distance is 2 for an

operand or is 1 for both operands40 = 5 ∗ 8

operand intrusions a digit of the result matches an operandfirst operand decade digit matches first operand 74←↩ 7 ∗ 8second operand unit digit matches second operand 68←↩ 7 ∗ 8consistency errorsunit consistency only unit digit is correct 76←↩ 56decade consistency only decade digit is correct 51←↩ 56off-by- errorsoff-by-±1 the result differs by x = 1 55 or 57off-by-±2 the result is off by −2 ≤ x ≤ 2 54, 55, 57, 58pattern errors swapped digits in the result 65confusion errors confusion with addition, subtraction and

division operations15 or 1

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201610

Page 11: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Model Structure (1/4)

Perturbation model

Bug library with 6 error types : operand, intrusion,consistency, off-by-±1 and off-by-±2, pattern,confusion with addition, subtraction, and divisionoperation

Unclassified errors

Correct (absence of misconception / error)

Clean up the data samples

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201611

Page 12: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Model Structure (2/4)

Minimal structural assumptions: Errors don’t dependon each other directly

But there are answers to questions that could belongto many error types, f.e. a student may answer thequestion 8 × 5 with a 41, which could be both aconsistency and off-by-+1 error.

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201612

Page 13: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Model Structure (3/4)

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201613

Page 14: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Model Structure (4/4)

The learner’s state of knowledge changes over timeeven during application useThe model represents our estimations about thestudent’s misconceptionsThe more data, the more certain we will be whicherror type is more dominant, how much, etc.“A particular person, when dealing with one-digitmultiplication problems, makes 20% operand errors,10% consistency errors” and so onUncover the similarities and the differences of thelearners

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201614

Page 15: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Learning the Model’s Parameters (1/3)

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201615

Page 16: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Learning the Model’s Parameters (2/3)

The dependencies define the structure of the modelThe quantification of this relationships is expressedby the parameters, which will be learned by the dataInitialisation assumption: non-informative uniformprior

1. Each error type and the unique correct option areequally likely

2. Given a specific error type, is there anypreference or tendency towards a particularanswer?

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201616

Page 17: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Learning the Model’s Parameters (3/3)

Expectation-maximization (EM) - Algorithm, becausethe “LearningState” random variable is hidden

Probabilistic programming library Figaro for Scalahttp://www.scala-lang.orghttps://www.cra.com/work/case-studies/figarohttps://www.cra.com/sites/default/files/pdf/Figaro Tutorial.pdfhttps://www.manning.com/books/practical-probabilistic-programming

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201617

Page 18: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Future Work (1/2)

Are there new types of errors (according to the modeland data)?

Identify groups of learners (stereotypes) with mixtureof Dirichlet distributions (“Learning State” of 20% ofour students has values near to a particular Dirichletdistribution, 30% equals a completely differentDirichlet distribution and so on)

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201618

Page 19: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Future Work (2/2)

Predict the answer and use this to decide what thenext learning issue will be

Probabilistic and maximum a posteriori (MAP)queries can be used to influence all kinds of factors ina learning application such as hints, helping notes,rearrangement of questions’ sequence

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201619

Page 20: Bayesian Modelling of Student Misconceptions in the one-digit Multiplication with Probabilistic Programming

http://elearning.tugraz.at

Thanks for your attention!Comments?

Behnam Taraghi, Anna Saranti, Robert Legenstein, Martin Ebner, Graz University of TechnologyEdinburgh, 29.04.201620