Upload
nantai
View
72
Download
0
Embed Size (px)
DESCRIPTION
Skills Diagnosis with Latent Variable Models. Topic 1: A New Diagnostic Paradigm. Introduction. Assessments should aim to improve, and not merely ascertain the status of student learning - PowerPoint PPT Presentation
Citation preview
Skills Diagnosis with Latent Variable Models
Topic 1:
A New Diagnostic Paradigm
• Assessments should aim to improve, and not merely ascertain the status of student learning
• For test scores to facilitate learning, they need to be interpretative, diagnostic, highly informative, and potentially prescriptive
• Most large-scale assessments are based on traditional unidimensional IRT models that only provide single overall scores
• These scores are useful primarily for ordering students along a continuum
Introduction
• Alternative psychometric models that can provide inferences more relevant to instruction and learning currently exist
• These models are called cognitive diagnosis models (CDMs)
• Alternatively, they are referred to as diagnostic classification models (DCMs)
• CDMs are multiple discrete latent variable models
• They are developed specifically for diagnosing the presence or absence of multiple fine-grained attributes (e.g. skills, cognitive processes or problem-solving strategies)
• Fundamental difference between IRT and CDM: A fraction subtraction example
• IRT: performance is based on a unidimensional continuous latent trait
• Students with higher latent traits have higher probability of answering the question correctly
7412 122
0.00
0.20
0.40
0.60
0.80
1.00
-3.5 -2.5 -1.5 -0.5 0.5 1.5 2.5 3.5
( 1| 0.8) 0.3P X
( 1| 1.2) 0.9P X
0.8 1.2
• Fundamental difference between IRT and CDM: A fraction subtraction example
• IRT: performance is based on a unidimensional continuous latent trait
• Students with higher latent traits have higher probability of answering the question correctly
• CDM: performance is based on binary latent attribute vector
• Successful performance on the task requires a series of successful implementations of the attributes specified for the task
7412 122
1( , , )K
7412 122
127
12161
1291
431
• Required attributes:
(1) Borrowing from whole
(2) Basic fraction subtraction (3) Reducing
• Other attributes:
(5) Converting whole to fraction
(4) Separating whole from fraction
• The response vector of examinee i will be denoted by ,
• The response vector contains J items, as in,
• The attribute vector of examinee i will be denoted by
• Each attribute vector or pattern defines a unique latent class
• Thus, K attributes define latent classes
Basic Elements and Notations of CDM
• Example: When , the total number of latent classes is
• Although arbitrary, we can associate the following attribute vectors with the following latent classes:
• Like IRT, CDM requires an binary response matrix as input
• Unlike IRT, CDM in addition requires a binary matrix called the Q-matrix as input
• The rows of the Q-matrix pertain to the items, whereas the columns the attributes
• The 1s in the jth row of the Q-matrix identifies the attributes required for item j
Basic CDM Input
Attribute
Item
(1)
Borrow from the whole
(2)
Basic fraction
subtraction
(3)
Reduce
(4) Separate
whole from
fraction
(5)
Convert whole to fraction
Examples of Attribute Specification
4 71. 2
12 12 1 1 1 0 0
3 42. 7 2
5 5 11 0 1 0
Attribute
Item
(1)
Borrow from the whole
(2)
Basic fraction
subtraction
(3)
Reduce
(4) Separate
whole from
fraction
(5)
Convert whole to fraction
Examples of Attribute Specification
13. 2
3
74. 3 2
8
• The goal of CDM is to make inference about the attribute vector
• The basic CDM output gives the (posterior) probability the examinee has mastered each of the attributes
• That is, we get
• For example, , indicates that we are quite certain that examinee has already mastered attribute 1
Basic CDM Output
• Each examinee gets a vector of posterior probabilities
• For reporting purposes, we may want to convert the probabilities into 0s and 1s
• We can use different rules for this conversion
• If ;
Otherwise,
Example:
• Each examinee gets a vector of posterior probabilities
• For reporting purposes, we may want to convert the probabilities into 0s and 1s
• We can use different rules for this conversion
• If ;
Otherwise,
• If ; or
If ;
Otherwise,
Example:
? – means we do not have sufficient evidence to conclude one way or the other