Upload
ann
View
49
Download
3
Embed Size (px)
DESCRIPTION
The Poisson-Gamma model for speed tests. Norman Verhelst Frans Kamphuis National Institute for Educational Measurement Arnhem, The Netherlands. The student monitoring system. Measurement of individual development Common scale Estimation of distribution (norms) - PowerPoint PPT Presentation
Citation preview
The Poisson-Gamma model for speed tests
Norman VerhelstFrans Kamphuis
National Institute for Educational Measurement Arnhem, The Netherlands
The student monitoring system
• Measurement of individual development – Common scale
• Estimation of distribution (norms) – Twice per grade (M3, E3,…,M8)
• Several subjects– Arithmetic– Reading comprehension– Technical reading
Two types of speed tests
• Basic observation is the time to complete a task– AVI cards
• Basic observation is the number of completed subtasks within the time limit– Tempotests (TT)– Three Minute Test (TMT)
Example tempotest (E4)
• Op de politieschool spelen ze ook rook koor een soort toneel
• Het lijkt wel wat op ‘politie en boefje spelen stelpen slepen’.
• Net zoals op de basisschool.
• Wat poe doe boe je bij een gevecht?
• Je pistool trekken?
• Nee, dat mag zomen zomaar zomer niet.
Example TMT
• Easy version– as– fee– oom– uur– zee– oor– …– poot (=150)
• Hard version– banden– geluid– tante– beker– kuiken– koffer– …– brandweerwagen
(=150)
Models
• Measurement model: Poisson– What is the relation between the (latent)
ability and the test performance?
• Structural model: Gamma– The distribution of the latent ability in one or
more populations? (M3, E3, M4,…,M8)
Measurement model: Poisson (1)
: observation (number read/number correct)
: student index
: task index
vix
v
i
( ; ) , ( 0,1,2,3, )!
vix
vi vivi
P x e xx
Measurement model: Poisson (2)
( ; ) , ( 0,1, 2,3, )!
vix
vi vivi
P x e xx
vi i v i
: time limit (in minutes)
: easiness of task (dimensionless)
: ability (#subtasks/minute)
i
i
v
i
Parameter estimation:incomplete design (JML)
1
statistics: en k
v vi vi i vi vii v
s d x t d x
1
normalisation: 1k
ii
vv
vi i ii
s
d
ii
i vi vv
t
d
Person parametersˆ ˆv vi i i
i
d
ˆˆv
v
v
s
ˆ |E
ˆˆ( )
ˆ ˆvv v
vv v v
sSE
is the corrected reading time (weights: )i
Design TMT
• 3 difficulty levels (1, 2, 3)
• For each level: three parallell versions (a, b, c)
• Each student participates twice: medio and end of same grade
• At each administration: 3 cards of levels 1, 2 and 3 (in that sequence)
• M3: only cards 1 and 2
voor de groepen 4-7medio eind
1 2 3 1 2 31 a a a b b b2 a a b b b c3 a a c b b a4 a b a b c b5 a b b b c c6 a b c b c a7 a c a b a b8 a c b b a c9 a c c b a a
10 b a a c b b11 b a b c b c12 b a c c b a13 b b a c c b14 b b b c c c15 b b c c c a16 b c a c a b17 b c b c a c18 b c c c a a19 c a a a b b20 c a b a b c21 c a c a b a22 c b a a c b23 c b b a c c24 c b c a c a25 c c a a a b26 c c b a a c27 c c c a a a
Two step procedure
• Estimate the task parameters σi
– JML = CML
• Estimate latent distribution while fixing the task parameters at their CML -estimate
Advantage
1 2 1 2
1 2 1 2
If and indep. Poisson with parameters en ,
then is Poisson distributed with parameter
X X
X X
[ ] ( )v vi v i i vi i
s s P P
Structural model:distribution of reading speed (θ)
1 ( ; , ) exp( )( )
g
( )E
2
( )Var
Marginal distribution of the sum score s
0
1
0
( | )
( )
( ) (
!
)
( )
s
P s
es
f s g d
e d
Negative Binomial(Gamma-Poisson)
( )( )
! ( ) ( )
s
s
sf s
s
p
1 p
( )( ) (1 )
! ( )ss
f s p ps
Negative binomial
1
0
( 1) ( )
( ) ( )( )
( ) ( )
s
j
sj
1
0( )
( ) (1 )!
s
j sj
f s p ps
EAP
| Gamma( , )s s
( | )s
E s
( | )s
SD s
Reliability
'SS p
Validation (tempo test)M4
0
5
10
15
20
25
25 50 75 100 125 150 175
Validation (tempo test)
0.00
0.25
0.50
0.75
1.00
25 50 75 100 125 150 175gobserveerde scores
exp(M4)
obs(M4)
exp(E4)
obs(E4)
Validation (TMT)
M3
0
5
10
15
20
25
30
0 50 100 150 200
Latent class model• Population consists of two latent classes
of size π and 1 - π respectively • The latent variable is gamma distributed in
each class• Parameters
– π– α1 en β1
– α2 en β2
• EM-algorithm
M3 (pi = 0.54)
0 20 40 60 80 100theta (words per minute)
class 1
class 2
mixture
Validation (TMT)
M3
0
5
10
15
20
25
30
0 50 100 150 200
Validation (TMT)
0.00
0.25
0.50
0.75
1.00
0 50 100 150 200 250aantal woorden gelezen
exp(M3)
obs(M3)
exp(E3)
obs(E3)
Norms (TMT)
0.00
0.25
0.50
0.75
1.00
0 20 40 60 80 100 120theta (= woorden per minuut)
M3 E3 M4 E4 M5 E5 M6 E6 M7 E7 M8
Thank you
Example: student vTask i dvi
1 0 8 0.93 -
2 1 8 1.11 8.88
3 0 6 0.85 -
4 1 6 1.05 6.30
5 0 5 1.09 -
δv : 15.18
i
ivi id i
122122 8.04 (subtasks/minute on a standard task)
15.18vvs
122( ) 0.73
15.18vSE
Problems
• SE(π) large
• Local maxima?
• Thick right tail of observations
• >2 classes?– Initial estimates
• Homogeneity of test material
• Local independence
Simulation E3
0
0.2
0.4
0.6
0.8
1
10 15 20 25 30 35 40
average class 1
siz
e c
las
s 1
real pi = 0.51; estimated pi = 0.93
0
200
400
600
800
1000
0 50 100 150 200 250score
cu
mu
lati
ve
fre
qu
en
cy Obs.
Exp.
Class 1 Class 2 Overall
Mean 28.15 44.07 35.99
SD 2.71 3.22 0.43
Averages (1000 replications)
Standard deviations (1000 rep.)
Class 1 Class 2 Overall
Mean 13.31 17.44 17.66
SD 2.21 1.68 0.47