Multiple-choice Questionand Item Analysis

ปรเมศวร เหลาสินชัย

4, 11 กรกฎาคม 2562

Outline1. Bloom’s taxonomy revisited2. Learning outcomes revisited3. Activity: Learning outcomes of each termite topic4. Activity: MCQs for each termite topic5. Activity: Distractor analysis6. Activity: Item difficulty index7. Activity: Item discrimination index8. Reliability9. Test composition10. Types of multiple-choice questions

2

3https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

2. Learning outcomes revisited

• Active verb• What should students be able to do?• May be taken from Bloom’s taxonomy sample verbs

• Learning content• What is the action applied to?• Try to be specific

4

3. LOs of each termite topic

Termite expert groups• ขอมูลทั่วไป• ประโยชน• การจัดการศัตรู• การจัดการสังคม• ผลตอสิ่งมีชีวิต• เคมีและการปองกันEach group may search for extra information

5

4. MCQs for each termite topic

• Each group uses Kahoot to create 5 MCQs• Try to cover all the learning objectives• Other groups take the test

6

5. Distractor analysis

• Order the choices by the numbers of students selecting them• Which choice should be selected most?• Which choice was selected by high/low achievers?• Which choices were not selected?• Do they indicate that something needs improvement?

• The question statement• The choices• Eliminate the item

7

6. Item difficulty index

• How easy an item is?• Normalized (0–1) average score of the item

• E.g. 30 from 40 students got it right• Item difficulty index is 30/40 or 0.75

• What are the item difficulty indices of your test?• Can be calculated for the whole test• For MCQ, what does it imply?

8

9https://www.dek-d.com/activity/26207/

7. Item discrimination index

• How well an item discriminates high from low performers?• May use

• Normalized difference between the high and low performers• Corrected item-total correlation

• Normalized difference• Divide students into 3 groups according to their performances on the test• Find the difference of the difficulty indices for the high and low groups

• Corrected item-total correlation• For each student, subtract the score of the item from the total score• Find the correlation between the item’s scores and the total scores

10

8. Reliability

• How consistent the test results are?• Correlation of the test results• Test-retest reliability

• Same test, same group, different times

• Parallel-form reliability• Different tests, same group, different times• Pre and post tests

• Internal consistency• Cronbach’s alpha (average covariance to average variance ratio)

11

Covariance matrix

12https://stats.seandolinar.com/making-a-covariance-matrix-in-r/

9. Test composition

Learning Outcome PercentLO1 xxLO2 xxLO3 xxLO4 xx

13

10. Types of MCQs

• Normal MCQ• Multi-tier MCQ

• Second tier may be the explanations• True/False• Matching• Multiple answers / All possibilities

• A student can select more than one choice or• Given 2 to 3 statements, the choices are which one(s) are correct

14