Mathematical Models for Data Analysis MARM Lecture 4 · 2009-09-02 · Analysis of ﬁt Item calibration and estimate of person’s abilities are justonly part of the complete analysisof

Mathematical Models for Data AnalysisMARM Lecture 4

Enrico Rogora.

NAIROBI , September1st 2009

Enrico Rogora. MARM Lecture 4

Analysis of fit

Item calibration and estimate of person’s abilities are just only part ofthe complete analysis of a sample of data.

We need also an analysis of how well data fit the RASCH MODEL.

Example of implausible pattern

Wrong answers to all easy questions. Right answers to all difficultquestions.


Analysis of fit

Item calibration and estimate of person’s abilities are just only part ofthe complete analysis of a sample of data.We need also an analysis of how well data fit the RASCH MODEL.




Analysis of fit





Analysis of fit





Knox Cube Text data

1 1 42 2 33 1 2 44 1 3 45 2 1 46 3 4 17 1 4 3 28 1 4 2 39 1 3 2 410 2 4 3 111 1 3 1 2 412 1 3 2 4 313 1 4 3 2 414 1 4 2 3 4 115 1 3 2 4 1 316 1 4 2 3 1 417 1 4 3 1 2 418 4 1 3 4 2 1 4


c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 c15 c16 c171 1 1 1 1 0 0 0 0 0 0 0 0 0 02 1 1 1 1 1 1 1 0 0 0 0 0 0 03 1 1 1 1 1 1 0 0 1 0 0 0 0 04 1 0 0 1 0 1 0 0 0 0 0 0 0 05 1 1 1 1 1 1 1 0 0 0 0 0 0 06 1 1 1 1 1 1 1 0 0 0 0 0 0 07 1 1 1 1 1 1 1 1 1 1 0 1 0 08 1 1 1 1 1 1 1 0 0 0 0 0 0 09 1 1 1 1 1 1 1 0 0 0 0 0 0 0

10 1 1 1 1 1 1 1 1 0 0 0 0 0 011 0 1 1 1 1 1 0 0 0 0 0 0 0 012 1 1 0 1 0 1 1 0 0 0 0 0 0 013 1 1 0 0 1 1 1 1 1 0 0 0 0 014 1 1 1 1 1 1 1 1 0 0 0 0 0 015 1 1 1 1 1 1 1 1 1 1 0 0 0 016 1 1 1 1 1 1 0 1 0 0 0 0 0 017 1 0 1 1 1 1 1 0 0 0 0 0 0 018 1 1 1 1 1 1 1 0 0 1 0 0 0 019 1 1 1 1 1 1 0 0 0 0 0 0 0 020 1 1 1 1 1 1 1 0 0 1 0 0 0 021 1 1 1 1 1 1 1 1 0 1 0 0 0 022 1 1 1 1 1 1 1 1 1 0 0 0 0 023 1 1 1 1 1 1 1 0 0 1 1 0 0 024 1 1 1 1 1 1 1 1 0 1 0 0 1 125 0 1 1 0 0 0 0 0 0 0 0 0 0 026 1 1 1 1 1 1 1 0 0 0 0 0 0 027 1 1 1 1 0 0 0 0 0 0 0 0 0 028 1 1 1 1 1 1 0 1 0 0 0 0 0 029 1 1 1 0 0 1 1 1 0 0 1 0 0 030 1 1 1 1 1 1 0 0 0 0 0 0 0 031 1 1 1 1 1 1 1 0 0 0 0 0 0 032 1 1 1 1 1 1 1 1 0 0 0 0 0 033 1 0 0 1 0 0 1 0 0 0 0 0 0 034 1 1 1 1 1 1 1 0 1 0 1 0 0 0


c4 c5 c7 c6 c9 c8 c10 c11 c13 c12 c14 c15 c16 c1725 0 1 0 1 0 0 0 0 0 0 0 0 0 04 1 0 1 0 1 0 0 0 0 0 0 0 0 0

33 1 0 1 0 0 0 1 0 0 0 0 0 0 01 1 1 1 1 0 0 0 0 0 0 0 0 0 0

27 1 1 1 1 0 0 0 0 0 0 0 0 0 011 0 1 1 1 1 1 0 0 0 0 0 0 0 012 1 1 1 0 1 0 1 0 0 0 0 0 0 017 1 0 1 1 1 1 1 0 0 0 0 0 0 019 1 1 1 1 1 1 0 0 0 0 0 0 0 030 1 1 1 1 1 1 0 0 0 0 0 0 0 0

2 1 1 1 1 1 1 1 0 0 0 0 0 0 03 1 1 1 1 1 1 0 0 0 1 0 0 0 05 1 1 1 1 1 1 1 0 0 0 0 0 0 06 1 1 1 1 1 1 1 0 0 0 0 0 0 08 1 1 1 1 1 1 1 0 0 0 0 0 0 09 1 1 1 1 1 1 1 0 0 0 0 0 0 0

13 1 1 0 0 1 1 1 1 0 1 0 0 0 016 1 1 1 1 1 1 0 1 0 0 0 0 0 026 1 1 1 1 1 1 1 0 0 0 0 0 0 028 1 1 1 1 1 1 0 1 0 0 0 0 0 029 1 1 0 1 1 0 1 1 0 0 1 0 0 031 1 1 1 1 1 1 1 0 0 0 0 0 0 010 1 1 1 1 1 1 1 1 0 0 0 0 0 018 1 1 1 1 1 1 1 0 1 0 0 0 0 014 1 1 1 1 1 1 1 1 0 0 0 0 0 032 1 1 1 1 1 1 1 1 0 0 0 0 0 020 1 1 1 1 1 1 1 0 1 0 0 0 0 021 1 1 1 1 1 1 1 1 1 0 0 0 0 022 1 1 1 1 1 1 1 1 0 1 0 0 0 023 1 1 1 1 1 1 1 0 1 0 1 0 0 034 1 1 1 1 1 1 1 0 0 1 1 0 0 015 1 1 1 1 1 1 1 1 1 1 0 0 0 0

7 1 1 1 1 1 1 1 1 1 1 0 1 0 024 1 1 1 1 1 1 1 1 1 0 0 0 1 1


c4 c5 c7 c6 c9 c8 c10 c11 c13 c12 c14 c15 c16 c1725 0 1 0 1 0 0 0 0 0 0 0 0 0 04 1 0 1 0 1 0 0 0 0 0 0 0 0 0

33 1 0 1 0 0 0 1 0 0 0 0 0 0 01 1 1 1 1 0 0 0 0 0 0 0 0 0 0

27 1 1 1 1 0 0 0 0 0 0 0 0 0 0

11 0 1 1 1 1 1 0 0 0 0 0 0 0 0

12 1 1 1 0 1 0 1 0 0 0 0 0 0 0

17 1 0 1 1 1 1 1 0 0 0 0 0 0 019 1 1 1 1 1 1 0 0 0 0 0 0 0 030 1 1 1 1 1 1 0 0 0 0 0 0 0 02 1 1 1 1 1 1 1 0 0 0 0 0 0 0

3 1 1 1 1 1 1 0 0 0 1 0 0 0 05 1 1 1 1 1 1 1 0 0 0 0 0 0 06 1 1 1 1 1 1 1 0 0 0 0 0 0 08 1 1 1 1 1 1 1 0 0 0 0 0 0 09 1 1 1 1 1 1 1 0 0 0 0 0 0 0

13 1 1 0 0 1 1 1 1 0 1 0 0 0 016 1 1 1 1 1 1 0 1 0 0 0 0 0 026 1 1 1 1 1 1 1 0 0 0 0 0 0 028 1 1 1 1 1 1 0 1 0 0 0 0 0 0

29 1 1 0 1 1 0 1 1 0 0 1 0 0 031 1 1 1 1 1 1 1 0 0 0 0 0 0 010 1 1 1 1 1 1 1 1 0 0 0 0 0 018 1 1 1 1 1 1 1 0 1 0 0 0 0 014 1 1 1 1 1 1 1 1 0 0 0 0 0 032 1 1 1 1 1 1 1 1 0 0 0 0 0 020 1 1 1 1 1 1 1 0 1 0 0 0 0 021 1 1 1 1 1 1 1 1 1 0 0 0 0 022 1 1 1 1 1 1 1 1 0 1 0 0 0 023 1 1 1 1 1 1 1 0 1 0 1 0 0 034 1 1 1 1 1 1 1 0 0 1 1 0 0 015 1 1 1 1 1 1 1 1 1 1 0 0 0 07 1 1 1 1 1 1 1 1 1 1 0 1 0 0

24 1 1 1 1 1 1 1 1 1 0 0 0 1 1


Unexpected patterns

We expect to find a run of success,

then a run of successes andfailures, leading finally to a run of failures as the items finally becometoo difficult.Some patterns seem to exceed this expectation. The mostimplausible are

Person 11Person 13Person 17Person 29


Unexpected patterns

We expect to find a run of success, then a run of successes andfailures,

leading finally to a run of failures as the items finally becometoo difficult.Some patterns seem to exceed this expectation. The mostimplausible are



Unexpected patterns

We expect to find a run of success, then a run of successes andfailures, leading finally to a run of failures as the items finally becometoo difficult.

Some patterns seem to exceed this expectation. The mostimplausible are



Unexpected patterns

We expect to find a run of success, then a run of successes andfailures, leading finally to a run of failures as the items finally becometoo difficult.Some patterns seem to exceed this expectation.

The mostimplausible are



Unexpected patterns

We expect to find a run of success, then a run of successes andfailures, leading finally to a run of failures as the items finally becometoo difficult.Some patterns seem to exceed this expectation. The mostimplausible are



Unexpected patterns


Person 11

Person 13Person 17Person 29


Unexpected patterns


Person 11Person 13

Person 17Person 29


Unexpected patterns


Person 11Person 13Person 17

Person 29


Unexpected patterns




Standard residuals

We want a systematic way to judge the degree of unexpectednessseen in these response patterns.

Recall that expectation and variance of xνi are,

E(Xνi) = πνi ≈ pνi = exp(bν − di)

V (Xν i) = E((Xν i − E(Xνi))2) = πνi(1− πνi)

Hence, the extimated standard residual of Xνi is

Zνi = (Xν i − pνi)/ (pνi(1− pνi))2.

As a rough but useful criterion for the fit of the data to the model wecan examine the extent to which

Zν i ∼ N(0,1) Z 2ν i = χ2

1.


Standard residuals

We want a systematic way to judge the degree of unexpectednessseen in these response patterns.Recall that expectation and variance of xνi are,


V (Xν i) = E((Xν i − E(Xνi))2) = πνi(1− πνi)




Zν i ∼ N(0,1) Z 2ν i = χ2

1.


Standard residuals



V (Xνi) = E((Xνi − E(Xνi))2) = πνi(1− πνi)




Zν i ∼ N(0,1) Z 2ν i = χ2

1.


Standard residuals







Zν i ∼ N(0,1) Z 2ν i = χ2

1.


Standard residuals





Zνi = (Xνi − pνi)/ (pνi(1− pνi))2.


Zν i ∼ N(0,1) Z 2ν i = χ2

1.


Standard residuals





Zνi = (Xνi − pνi)/ (pνi(1− pνi))2.


Zνi ∼ N(0,1) Z 2νi = χ2

1.


Measure of unexpectedness

Each of the random variables X = Xνi assumes only two values 0and 1. The corresponding standardized values depend on β and δ.

The estimate for the standardized value of 0 is

0− p

((1− p)p)1/2 = −√

p1− p

= −exp12

(b − d).


1− p

((1− p)p)1/2 = −

√1− p

p= −exp

12

(d − b).

We have UNEXPECTED answers in two cases



Each of the random variables X = Xνi assumes only two values 0and 1. The corresponding standardized values depend on β and δ.The estimate for the standardized value of 0 is

0− p

((1− p)p)1/2 = −√

p1− p

= −exp12

(b − d).


1− p

((1− p)p)1/2 = −

√1− p

p= −exp

12

(d − b).





0− p

((1− p)p)1/2 = −√

p1− p

= −exp12

(b − d).


1− p

((1− p)p)1/2 = −

√1− p

p= −exp

12

(d − b).





0− p

((1− p)p)1/2 = −√

p1− p

= −exp12

(b − d).


1− p

((1− p)p)1/2 = −

√1− p

p= −exp

12

(d − b).



Unexpected answers.

b − d > 0 and X = 0

We get an unexpected answer ifthe ability is greater than the difficultybut we get a wrong answer. We have.

Z = −exp b − d2

Z 2 = exp(b − d)

b − d < 0 and X = 1We get an unexpected answer if the difficulty is greater than theability but we get a wrong answer. We have.

Z = −exp d − b2

Z 2 = exp(d − b)


Unexpected answers.

b − d > 0 and X = 0We get an unexpected answer ifthe ability is greater than the difficultybut we get a wrong answer. We have.

Z = −exp b − d2

Z 2 = exp(b − d)


Z = −exp d − b2

Z 2 = exp(d − b)


Unexpected answers.


Z = −exp b − d2

Z 2 = exp(b − d)


Z = −exp d − b2

Z 2 = exp(d − b)


Unexpected answers.


Z = −exp b − d2

Z 2 = exp(b − d)


Z = −exp d − b2

Z 2 = exp(d − b)


Unexpected answers.


Z = −exp b − d2

Z 2 = exp(b − d)


Z = −exp d − b2

Z 2 = exp(d − b)


Significance of a measure of unexpectednessSince Xνi ∼ N (0,1), we can judge statistical significance of xν,i byrelating it to N (0,1).

-6 -4 -2 0 2 4 6

0.0

0.1

0.2

0.3

0.4

x

y

Numbers with absolute values less than 3, e.g. the x-value of thecontinuous vertical line, has no statistical significance.Numbers with absolute values greater than 3, e.g. the x-value of thedotted vertical line, has statistical significance.



-6 -4 -2 0 2 4 6

0.0

0.1

0.2

0.3

0.4

x

y




-6 -4 -2 0 2 4 6

0.0

0.1

0.2

0.3

0.4

x

y

Numbers with absolute values less than 3, e.g. the x-value of thecontinuous vertical line, has no statistical significance.

Numbers with absolute values greater than 3, e.g. the x-value of thedotted vertical line, has statistical significance.



-6 -4 -2 0 2 4 6

0.0

0.1

0.2

0.3

0.4

x

y



Misfit statistics.

Improbability of the response

ρ = 11+z2

0when b − d > 0, i.e. when zero is the unexpected

answer.ρ = 1

1+z21

when b − d < 0, i.e. when one is the unexpectedanswer.

In both cases ρ is the probability to give the unexpected answer.

Relative efficiency of the observation

The information given by an item about the interaction β − δ ismaximal when β = δ. The number I = 400ρ(1− ρ). gives the fractionof such information for any value β − δ.

Items needed to have maximal precision

The number L = 1000/I is the number of less efficient itemsnecessary to match the precision of 10 right on target items .


Misfit statistics.


ρ = 11+z2


answer.

ρ = 11+z2

1when b − d < 0, i.e. when one is the unexpected

answer.







Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








Misfit statistics.


ρ = 11+z2


answer.ρ = 1

1+z21








MisfittingSet z2

ν,i = 0.

Person misfitting

Let f = L− 1 and v =∑

i z2ν i ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)

The significant person misfit behaviors come from values of t whichare bigger than 3. Negative values do not count.

Item misfitting

Let f = N − 1 and u =∑ν z2

ν i ∼ χ2f . Then

t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)

The significant item misfit behaviors come from values of t which arebigger than 3. Negative values do not count. We may improve datafitting by discarding items and/or persons.


MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting



t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting



t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting



t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting



t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting


νi ∼ χ2f . Then

t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting


νi ∼ χ2f . Then

t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)

The significant item misfit behaviors come from values of t which arebigger than 3. Negative values do not count.

We may improve datafitting by discarding items and/or persons.


MisfittingSet z2

ν,i = 0.

Person misfitting


i z2νi ∼ χ2

f . Then

t = (log(v) + v − 1)/√

f/8 ∼ N (0,1)


Item misfitting


νi ∼ χ2f . Then

t = (log(u) + u − 1)/√

f/8 ∼ N (0,1)



Variable construction

After item calibration and model fit analysis one can turn to Variableconstruction.

We need to check wether our calibrated items spread out in a waythat shows a coherent and meaningful direction.

Notice that if two items are not well separated by their standarderrors, they do not define a meaningful direction, hence the spread ofitems difficulties need to substantially greater than the standard errorof their estimates.












KCT variable

Item Tapping calibration Standard error4 1-3-4 -4.2 0.85 2-1-4 -3.6 0.77 1-4-3-2 -3.6 0.76 3-4-1 -3.2 0.69 1-3-2-4 -3.2 0.68 1-4-2-3 -2.2 0.6

10 2-4-3-2 -1.5 0.511 1-3-1-2-4 0.8 0.513 1-4-3-2-4 1.9 0.512 1-3-2-4-3 2.1 0.614 1-4-2-3-4-1 3.2 0.715 1-3-2-4-1-3 4.6 1.116 1-4-2-3-1-4 4.6 1.117 1-4-3-1-2-4 4.6 1.1


KCT variable (II)

Most of the person fall in the center of the test where we have a largegap in test items.

If we want to discriminate better among the persons with mediumability we need to add middle range difficulty items.

Careful reconsideration of items is needed in order to prepare newitems that we expect in the middle range.

When calibrating the items in extended tests, one needs to check thattwo independent estimates of each common item are statisticallyequivalent .


KCT variable (II)






KCT variable (II)






KCT variable (II)






Comparing independent estimates

The distribution of the difference between two independent estimatesof difficulty parameters in a Rasch model is such that

ti12 = (di1 − di2/(s2

i1 + s2i2)∼ N (0,1)

where√

s2i1 + s2

i2 estimates the expected standard error of thedifference between the estimates di1, di2 of the same parameter δi .




ti12 = (di1 − di2/(s2

i1 + s2i2)∼ N (0,1)

where√

s2i1 + s2





ti12 = (di1 − di2/(s2

i1 + s2i2)∼ N (0,1)

where√

s2i1 + s2



Further topics

Complete analysis of fit

Designing testsAdaptive testsMeasurement under different testsData BanksQuality control


Further topics

Complete analysis of fitDesigning tests

Adaptive testsMeasurement under different testsData BanksQuality control


Further topics

Complete analysis of fitDesigning testsAdaptive tests

Measurement under different testsData BanksQuality control


Further topics

Complete analysis of fitDesigning testsAdaptive testsMeasurement under different tests

Data BanksQuality control


Further topics

Complete analysis of fitDesigning testsAdaptive testsMeasurement under different testsData Banks

Quality control


Further topics

Complete analysis of fitDesigning testsAdaptive testsMeasurement under different testsData BanksQuality control


Bibliografy

Andrich D., Rasch models for measurement, Sage Publications,1988.

Wright B., Test Design, Rasch Measurement, Mesa, 1979.


Documents

Mathematical Models for Data Analysis MARM Lecture 4 · 2009-09-02 · Analysis of ﬁt Item calibration and estimate of person’s abilities are justonly part of the complete analysisof