MC Item Writing Workshop

Nursing Workshop 2/12/2013

Educational

TESSERACT


Multiple Choice Item Writing Workshop

Doug Mauger EdDSchool Of Instructor [email protected]


Workshop Agenda

• Introductions

• Overview

• Perspectives and Attitudes

• Stakeholders

• Validity and Reliability

• Alignment

• Classifying skills and outcomes

• Best Practices in MC item writing

• Table of Specification or Blueprint

• Take Away

• Next Workshop


Introductions

• Name:

• What topics do you teach?

• How long have you been teaching?

• One thing you know about Item writing

• What do you want to get out of the workshop?


Workshop Agenda

• Introductions

• Overview

• Perspectives and Attitudes• Stakeholders

• Valdity and Reliability

• Alignment




• Take Away

• Next Workshop


Put each rider on the top

of a horse



of a horse



of a horse



of a horse



of a horse




What are your attitudes to creating exams?


Workshop Agenda

• Introductions

• Overview


• Stakeholders• Validity and Reliability

• Alignment




• Take Away

• Next Workshop


Stakeholders Gathering


Workshop Agenda

• Introductions

• Overview


• Stakeholders

• Validity and Reliability• Alignment




• Take Away

• Next Workshop


Instrument……Confidence


There were flaws in 46% of the total questions, with the number of flaws per test ranging from 36% to 65%. In a sample of examinations distributed to first and second year medical students(Downing, 2005)

Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Advances in Health Science Education 10 (2), 133–143.

Points to Ponder


Process ValidityActivity


Validity and Reliability


Reliable but Not Valid


NOT Valid and NOT Reliable


Valid and Reliable


Validity Content

Predictive

Construct

Consequences

Process


Reliable Test – Retest

Alternate Form

Inter Rater


Types of ValidityActivity


1. Process, Content Validity (inadequate sample)

2. Process Validity (stress)

3. Content Validity (inadequate preparation)

4. Predictive Validity (inconsistent results)

5. Consequence Validity (induced anxiety)

6. Reliability/Process Validity (possible bias)


7. Process Validity (students not able to provide best performance)

8. Content Validity (different instructor’s test reflects different instruction)

9. Construct Validity (improper form of assessment)

10. Content Validity/Alignment (instruction and assessment do not match)


Why is Validity and Reliability important to you as a professional?


Workshop Agenda

• Introductions

• Overview


• Stakeholders

• Validity and Reliability

• Alignment• Classifying skills and outcomes



• Take Away

• Next Workshop


Alignment

Content and Assessment

Alignment

Are Critical

For Valid

Assessment

Intended

Curriculum

WHAT we plan

on teaching

Implemented

Curriculum

HOW we teach

the curriculum

Achieved

CurriculumAssessment

Evidence of

Student

Learning

Assessment

Perfect Alignment

Co

mp

lex

ity o

f c

on

ten

t

HowWhat

Outcomes: Intended Implemented Achieved

Assessment

Scenario 1

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 2

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 3

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 4

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 5

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 6

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 7

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Scenario 8

Co

mp

lex

ity o

f c

on

ten

tHowWhat

Assessment

Perfect Alignment

Co

mp

lex

ity o

f c

on

ten

tHowWhat


How has Alignment

Impacted You?


Workshop Agenda

• Introductions

• Overview


• Stakeholders


• Alignment

• Classifying skills and outcomes• Best Practices in MC item writing


• Take Away

• Next Workshop



Blooms Taxonomy revised


Sorting skills/outcomes

fromSimple to Complex


Blooms Taxonomy revised


Remembering/Understanding Level

Describes the circulation through the heart

Defines the term systole


Application (AP)theories in providing care to clients (e.g., a

Application Level

Apply the concepts of aging in developing interventions for the elderly


Blooms Taxonomy revisedCritical Thinking Level

Develop a plan for delivering services to elderly persons who are homebound

Evaluate nursing research based on predetermined criteria

Analyzes the organizational structure of the community health agency and its impact on client services.


Sorting skills/outcomes

into 3 classifications


Factual KnowledgeThe basic elements students must know to be acquainted with a discipline or solve problems in it.

Conceptual KnowledgeThe interrelationships among the basic elements within a larger structure that enable them to function together.

Procedural KnowledgeHow to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.

Meta-Cognitive KnowledgeKnowledge of cognition in general as well as awareness and knowledge of one's own cognition

Four Knowledge Dimensions


Verbs



Identify skills/outcomes

By

Knowledge Dimension & Cognitive Process


Writing a MC ItemWhat is the biggest challenge for you in sorting outcomes?


Workshop Agenda

• Introductions

• Overview


• Stakeholders


• Alignment


• Best Practices in MC item writing• Table of Specification or Blueprint

• Take Away

• Next Workshop


Writing a MC Item


Writing a MC Item


MC Item Anatomy


1. Selecting the competency/outcome

2. Construct the stem

3. Write plausible distractors

4. Peer edit

5. Item analysis

Steps to writing a MC Item item


ATOA and NOTA


Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option.

Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

Constructing a MC ItemPoints to Ponder

http://www.georgiasouthern.edu/ijsotl



Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

Constructing a MC Item




2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint submitted to Advances in Health Science Education

The present study examined the effect of NOTA on multiple choice questions in a large classroom setting. Specifically, we manipulated the place- ment of NOTA across versions of the test such that, for each of 20 experimental questions, NOTA was either not present, replaced the key, or replaced the high, medium, or low frequency distractors respectively. We found that NOTA sig-nificantly increased the difficulty of multiple choice questions only when it was used as the key. We found no effect of NOTA replacing any of the distractors, regardless of their effectiveness. Addi-tionally, we found no significant effect of NOTA on discrimination scores, regardless of its place- ment.

Constructing a MC ItemPoints to Ponder


Constructing a MC Item


Correct Answer

Best Answer

Correct or Best



Content Free Test


Varieties of MC Items

1. Single Correct Answer

2. Multiple Response

3. Combined Response

4. Multiple Response


Writing a MC Item

1. Design each item to measure an important learning outcome

2. Present a single clearly formulated problem in the stem of the item

3. State the stem of the item in simple, clear language

4. State the stem in positive form wherever possible

5. Word the alternatives clearly and concisely

6. Keep the alternatives mutually exclusive


7. Keep the alternatives homogeneous in content

8. Keep the alternatives free from clues as to which response is correct

9. Avoid the alternatives AOTA and NOTA

10. Use as many functional distractors as are feasible.

11. Include one and only one correct or clearly best answer for each item

Writing a MC Item


Writing a MC Item

12. Present the answer in each of the alternative positions approximately an equal number of times, in a random order

13. Layout the items in a clear and consistent manner

14. Use proper grammar, punctuation, and spelling

15. Avoid using unnecessarily difficult vocabulary

16. Analyze the effectiveness of each item after each administration of the test.

Writing a MC Item


Rules Broken in the

Content Free Test


Writing a MC Item

Item writing……


Workshop Agenda

• Introductions

• Overview


• Stakeholders


• Alignment



• Table of Specification or Blueprint • Take Away

• Next Workshop


Blueprint or TOS


Remembering/Understanding Level


Application (AP)theories in providing care to clients (e.g., a

Application Level


Blooms Taxonomy revisedCritical Thinking Level


Blueprint or TOS


Workshop Agenda

• Introductions

• Overview


• Stakeholders


• Alignment




• Take Away• Next Workshop


Writing a MC Item

WhatSo what Now what


Writing a MC Item


Workshop Agenda

• Introductions

• Overview


• Stakeholders


• Alignment




• Take Away

• Next Workshop


Writing a MC ItemThank You


References

http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxonomy%20Question%20Stems.pdf

Anderson, Lorin W. & Krathwohl, David R. (2001). A Taxonomy for Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy. New York. Longman Publishing.

Documents

MC Item Writing Workshop