Embed Size (px)
DESCRIPTION
Michael C. Rodriguez University of Minnesota CCSSO’s National Conference on Student Assessment June 2008. Modified Achievement Tests for Students with Disabilities: Distractor Analysis. Item Writing: Assessment Building Blocks. Item writing is an art and a science - PowerPoint PPT Presentation
Citation preview
Modified Achievement Testsfor Students with Disabilities:
Distractor Analysis
Michael C. RodriguezUniversity of Minnesota
CCSSO’s National Conference on Student AssessmentJune 2008
Item Writing: Assessment Building Blocks
• Item writing is an art and a science• Item writing requires supervised training• Empirical research on item writing began in 1920s• Multiple-choice items are capable of measuring a
wide range of content and cognitive domains in a short time with a high level of reliability
• Not all students are able to perform on today’s multiple-choice tests without accommodations or modifications
Item Modification
• Much of the language surrounding test item modification suggests that the goal is to make items “easier”.
• But making items easier doesn’t necessarily improve measurement.
• Elements of Universal Design provide a model for making appropriate modifications.
• Empirical research on item writing provides an evidence basis for item modifications.
Measurement & Access
• Test items provide opportunities for students to display construct-relevant knowledge, skills, and abilities.
• From a measurement perspective, item modifications should be done to– Provide access to the item for all students– Improve measurement of the construct
• The hypothesis is: By providing greater access to each item, we improve measurement.
Number of Options
• Less time is needed to prepare 2 plausible distractors than 3 or 4 distractors
• More 3-option items can be administered within the same time limit than 4 or 5-option items, improving content coverage
• Evidence suggests no significant reduction in test item or test score quality by reducing the number of options
Sample Item
PesticidesIn the late 1980s, farmers began to use a pesticide to control insects that harmed their cotton crops. This problem was solved. However, an insect group that pollinated the corn crops was also injured. Without pollination the corn kernels did not fully develop. This affected the corn harvest on which the farm families had come to depend. What is not mentioned as one effect of pesticide usage?
A. soil contaminationB. destruction of pestsC. destruction of friendly insectsD. crop losses
PesticidesIn the late 1980s, farmers began to use a chemical pesticide. It was used to control insects that harmed their cotton crops. This solved one problem, but caused another. An insect group that pollinated the corn crops was also harmed by the pesticide. Without pollination the corn kernels did not fully develop. This decreased the corn harvest.
What is NOT mentioned as one effect of using chemical pesticides?
A. destruction of the soil
B. destruction of pests
C. destruction of friendly insects
CAAVES Reliability Results
• All forms retained an acceptable level of reliability – for 39 item tests– .85 for Mathematics– .90 for Reading
• Significant Group x Condition Interaction– Test score reliability depended on the Group
(Disabled v. Nondisabled students) and Condition (Modified v. Unmodified forms) combination
Group x Condition Interaction
Disabled/Nondisabled differences• Mathematics
– Modified Forms: .025– Unmodified Forms: .027
• Reading– Modified Forms: .029– Unmodified Forms: .020
Group x Condition Interaction
Subject Condition Nondisabled – Disabled
Nondisabled – Eligible
Mathematics Modified .025 .046Unmodified .027 .021
Reading Modified .029 .058Unmodified .020 .039
The magnitude of differences in Reliability differed by Group x Condition combination, but appeared to be a function of the “Eligible” group
Condition x Group InteractionUNUEUDRNRERDMNMEMD
Co
eff
icie
nt
Alp
ha
fo
r 39
-ite
m T
es
ts
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Mathematics
Condition x Group InteractionUNUEUDRNRERDMNMEMD
Co
eff
icie
nt
Alp
ha
fo
r 39
-ite
m T
es
ts
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Reading
Correct Option Location
Content All Items Excluding D to CComp Correlation .454 .569 Sig. (2-tailed) .044 .017
N 20 17
Data Correlation -.450 -.453 Sig. (2-tailed) .053 .059
N 19 18
Number Correlation .225 .295 Sig. (2-tailed) .341 .235
N 20 18
Vocab Correlation .253 .244 Sig. (2-tailed) .296 .346
N 19 17
Spearman’s Rho Correlations between Change in Location and Difference in Item p-value
change1.000.500.00-0.50-1.00-1.50-2.00
dif
f
0.15
0.10
0.05
0.00
-0.05
-0.10
subject: d
change2.001.000.00-1.00-2.00
dif
f
0.30
0.20
0.10
0.00
-0.10
-0.20
subject: v
Issues related to Access
• Fewer options reduce the cognitive load• More options result in exposing additional
aspects of the domain to students – possibly providing clues to other questions
• More options can introduce irrelevant aspects of the domain
Method of Option Deletion
• Examining two common deletion methods (distractor functioning and random deletion)
• Going from 5 to 3 options: – No relation between deletion method and item difficulty
or discrimination (effect homogeneity)– No change in reliability for deletion of ineffective
distractors (.006)– Random deletion resulted in a reduction in reliability of .06
Improving Diagnostic Information
• Distractors that are written to be plausible should contain common errors or misconceptions
• Distractor analysis provides information regarding the kinds of errors or misconceptions held by students
• No reason, psychometrically, to have the same number of options for every item
ReferencesRodriguez, M.C. (2005). Three options are optimal
for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13.
Haladyna, T.M., Downing, S.M., & Rodriguez, M.C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334.
