Selected-Response Test Items

Multiple-Choice (MC) Multiple-Choice (MC) and and

Multiple-Response (MR) Multiple-Response (MR)

Dr. Belal Hijji, RN, PhDDr. Belal Hijji, RN, PhD

22/05/201122/05/2011

2

Learning OutcomesLearning Outcomes

At the end of this presentation, participants will be able to:At the end of this presentation, participants will be able to:• Realise the wide gap between universities in the ME and Realise the wide gap between universities in the ME and

those in the developed countries in relation to studies those in the developed countries in relation to studies focusing on educational measurement.focusing on educational measurement.

• Discuss the advantages and disadvantages of multiple-Discuss the advantages and disadvantages of multiple-choice items.choice items.

• Identify the parts of a multiple-choice item.Identify the parts of a multiple-choice item.• Discuss the challenges that face nursing educators when Discuss the challenges that face nursing educators when

writing multiple-choice items.writing multiple-choice items.• Identify the rules and principles of writing a MC item’s Identify the rules and principles of writing a MC item’s

stem and alternatives. stem and alternatives.

333

Rationale for the Presentation

It is known that nursing educators from the United States of

America had problems with constructing multiple-choice

items. In a study by Masters et al. (2001), 2,233 minor and

major violations of item-writing guidelines were identified.

Our question would then be:

What is it like when a nursing educator with English as a second language embarks on constructing multiple-choice

items without prior training?

444

Rationale for the Presentation (Continued ….)

What is also known is that studies focusing on educational

measurement were published from many countries and across

various disciplines (Ellsworth et al., 1990; Hansen, 1997;

Masters et al., 2001; Jozefowicz et al., 2002; Downing, 2005;

Shizuka et al., 2006; Tarrant et al., 2006; Tarrant et al., 2009).

However, similar publications from our region are lacking,

although the need for this type of studies is more visible.

Our question would then be:Should not we investigate the quality of our measurement

practice as a critical element of academic performance?

5

Advantages of Multiple-Choice ItemsAdvantages of Multiple-Choice Items

• Knowledge of facts, specific information, and principlesKnowledge of facts, specific information, and principles• Definitions of termsDefinitions of terms• Understanding of contentUnderstanding of content• Application of concepts in clinical situationsApplication of concepts in clinical situations• Analysis of data and clinical situationsAnalysis of data and clinical situations• Comparison and selection of varied interventionsComparison and selection of varied interventions• Judgments and decisions about actions to take in clinical Judgments and decisions about actions to take in clinical

situationssituations

6

Disadvantages of Multiple-Choice ItemsDisadvantages of Multiple-Choice Items

• These items are difficult to construct at higher cognitive These items are difficult to construct at higher cognitive levels. As such, many of them are written at the lower levels. As such, many of them are written at the lower cognitive levels.cognitive levels.

• Teachers often experience difficulty developing plausible Teachers often experience difficulty developing plausible distractors.distractors.

• It is often difficult to identify only one correct answer for It is often difficult to identify only one correct answer for an item.an item.

• As a result, these items are time-consuming to construct.As a result, these items are time-consuming to construct.• What would further compound these disadvantages is the What would further compound these disadvantages is the

fact that, in our region, multiple-choice items could be fact that, in our region, multiple-choice items could be written by nursing educators whose level of English written by nursing educators whose level of English language proficiency is inadequate. language proficiency is inadequate.

7

Parts of Multiple-Choice ItemParts of Multiple-Choice Item

An early and common sign An early and common sign STEM in form of STEM in form of of pregnancy is: of pregnancy is: incomplete statementincomplete statement

Options or AlternativesOptions or Alternativesa. amenorrheaa. amenorrhea AnswerAnswerb. morning sickness b. morning sickness DistractorDistractorc. spotting c. spotting DistractorDistractord. tenderness of the breasts d. tenderness of the breasts DistractorDistractor

8

What is the Major Challenge for Nursing What is the Major Challenge for Nursing Educators in ME universities?Educators in ME universities?

• Nursing educators should demonstrate their ability to adhere Nursing educators should demonstrate their ability to adhere to the various principles and rules of writing each of the to the various principles and rules of writing each of the above-mentioned parts. This is a critical issue for them to above-mentioned parts. This is a critical issue for them to infer that the scores a test yields are reliable and infer that the scores a test yields are reliable and meaningful, and is a true reflection of students’ knowledge meaningful, and is a true reflection of students’ knowledge and abilities. and abilities.

• We need to look at the parts as one unit; any violation of the We need to look at the parts as one unit; any violation of the said rules in any part may result in a weakness that said rules in any part may result in a weakness that increases the chances for students to guess the right answer. increases the chances for students to guess the right answer.

9

What are the Rules and Principles to Consider What are the Rules and Principles to Consider When Writing MCQWhen Writing MCQ

(Stem)(Stem)

1.1. The stem should present clearly and explicitly the problem The stem should present clearly and explicitly the problem to be solved. Options should be similar. For example:to be solved. Options should be similar. For example:

The causes of cataracts include:The causes of cataracts include:a. aging.*a. aging.*b. arteriosclerosis.b. arteriosclerosis.c. hemorrhage.c. hemorrhage.d. iritis.d. iritis.

A violation of this rule is provided next.A violation of this rule is provided next.*: Correct answer*: Correct answer

10

Cataracts:Cataracts:a. are painful.a. are painful.b. may accompany coronary artery disease.b. may accompany coronary artery disease.c. occur with aging.*c. occur with aging.*d. result in tunnel vision.d. result in tunnel vision.

The stem of the question does not clearly present the problemThe stem of the question does not clearly present the problemassociated with cataracts that the alternatives address. In associated with cataracts that the alternatives address. In

addition,addition,the options are dissimilar.the options are dissimilar.

11

2.2. The stem should be brief including ONLY necessary The stem should be brief including ONLY necessary information. An example showing a violation of this rule information. An example showing a violation of this rule follows.follows.

You are caring for an elderly man who lives alone butYou are caring for an elderly man who lives alone but

has frequent visits from his daughter. He has congestivehas frequent visits from his daughter. He has congestive

heart failure and some shortness of breath. Your patientheart failure and some shortness of breath. Your patient

was told recently that he has cataracts.was told recently that he has cataracts.

The causes of cataracts include:The causes of cataracts include:

a. aging.*a. aging.*

b. arteriosclerosis.b. arteriosclerosis.

c. hemorrhage.c. hemorrhage.

d. iritis.d. iritis.

The background information about the patient is irrelevant The background information about the patient is irrelevant to the problem addressed.to the problem addressed.

12

3.3. Avoid inserting information in the stem for instructional Avoid inserting information in the stem for instructional purposes. The goal of testing is not to teach new Information:purposes. The goal of testing is not to teach new Information:

Cataracts are an opacity of the lens or capsule of the eye leading toCataracts are an opacity of the lens or capsule of the eye leading toblurred and eventual loss of vision. The causes of cataractsblurred and eventual loss of vision. The causes of cataractsinclude:include:a. aging.*a. aging.*b. arteriosclerosis.b. arteriosclerosis.c. hemorrhage.c. hemorrhage.d. iritis.d. iritis.

13

4.4. Avoid using negatives, particularly double negatives, including Avoid using negatives, particularly double negatives, including words such as “no,” “not,” and “except.” In reality, however, not words such as “no,” “not,” and “except.” In reality, however, not all nursing educators have this knowledge (see below)all nursing educators have this knowledge (see below)

All of the following are realities about health education except? All of the following are realities about health education except?

a.a. It does It does not need special skills need special skills b.b. It depends on theories of learningIt depends on theories of learningc.c. It should be planned in participation with the learnerIt should be planned in participation with the learnerd.d. It is a continuing education that can happen in many settingsIt is a continuing education that can happen in many settings

((From a second exam))

All of them are an example for the Biological dimension except.All of them are an example for the Biological dimension except. a. genetica. genetic b. sexb. sex c. geographyc. geography ((From a final examFrom a final exam))

14

5.5. The stem and alternatives that follow should be consistent The stem and alternatives that follow should be consistent grammatically. If not, clues may be provided as to the grammatically. If not, clues may be provided as to the correct or incorrect responses. A consistent verb form should correct or incorrect responses. A consistent verb form should be used with the alternatives. For example:be used with the alternatives. For example:

Your patient is undergoing a right carotid endarterectomy. WhichYour patient is undergoing a right carotid endarterectomy. Which

pre-operative information would be most important to collect as apre-operative information would be most important to collect as a

baseline for the early recovery period? Her ability to:baseline for the early recovery period? Her ability to:

a. follow movements with her eyesa. follow movements with her eyes

b. move all four extremities*b. move all four extremities*

c. rotating her head from side to sidec. rotating her head from side to side

d. swallow and gagd. swallow and gag

15

6. Avoid ending stems with “a” or “an” because these often provide grammatical clues as to the option to select. It is usually easy to rephrase the stem to eliminate the “a” or “an.” For instance,

Narrowing of the aortic valve in children occurs with an:

a. aortic stenosis.*

b. atrial septal defect.

c. coarctation of the aorta.

d. patent ductus arteriosus.

16

Real-life examples of multiple violations, including the above rule, came from a nurse educator-constructed final exam (See below).

''any physiological or psychological factors necessary for a healthy

existence '' is a.

a. adaptation

b. human needs

c. concepts

Feelings toward a person, object or idea is an.

a. values

b. attitudes

c. ethics

Do other nurse educators in other ME universities have similar problems?

17

Alternatives (Options)Alternatives (Options)

1.1. The number of words included in each option should be The number of words included in each option should be consistent in length. Frequently the correct answer is the consistent in length. Frequently the correct answer is the longest and this may the testwise student to realize that it is the longest and this may the testwise student to realize that it is the correct answer. For example, correct answer. For example,

You are assessing an emaciated 14-year-old girl. Her motherYou are assessing an emaciated 14-year-old girl. Her mother

describes the following changes: resistance to eating and 20-lb.describes the following changes: resistance to eating and 20-lb.

weight loss over the last 6 weeks. It is most likely that the patientweight loss over the last 6 weeks. It is most likely that the patient

resists eating for which of the following reasons?resists eating for which of the following reasons?

a. Complains of recurring nausea.a. Complains of recurring nausea.

b. Describes herself as “fat all over” and fearful of gaining weight.*b. Describes herself as “fat all over” and fearful of gaining weight.*

c. Has other GI problems.c. Has other GI problems.

d. Seeks her mother’s attention.d. Seeks her mother’s attention.

The correct answer can be shortened to: Is fearful of gaining weight.The correct answer can be shortened to: Is fearful of gaining weight.

18

2.2. The options should have the same number of parts. In the The options should have the same number of parts. In the following example, including two causes in option “a” following example, including two causes in option “a” provides a clue to the answer. Revising that option to only provides a clue to the answer. Revising that option to only “aging” avoids this.“aging” avoids this.

Causes of cataracts include:Causes of cataracts include:

a. aging and steroid therapy.*a. aging and steroid therapy.*

b. arteriosclerosis.b. arteriosclerosis.

c. hemorrhage.c. hemorrhage.

d. iritis.d. iritis.

19

3.3. The alternatives should be consistent grammatically. The answer The alternatives should be consistent grammatically. The answer and distractors should be similar in structure and terminology. and distractors should be similar in structure and terminology. Without this consistency, the test-taker may be clued to the correct Without this consistency, the test-taker may be clued to the correct response or know to eliminate some of the options without being response or know to eliminate some of the options without being familiar with the content. See below how the correct answer familiar with the content. See below how the correct answer differs grammatically from the others:differs grammatically from the others:

You are making a home visit with a new mother who is breastfeeding.You are making a home visit with a new mother who is breastfeeding.

She tells you that her nipples are cracked and painful. Which of theShe tells you that her nipples are cracked and painful. Which of the

following instructions should be given to the mother?following instructions should be given to the mother?

a. Put the entire areola in the baby’s mouth during feeding.*a. Put the entire areola in the baby’s mouth during feeding.*

b. The baby should be fed less frequently until the nipples are healed.b. The baby should be fed less frequently until the nipples are healed.

c. There is less chance of cracking if the nipples are washed daily c. There is less chance of cracking if the nipples are washed daily with soap.with soap.

d. Wiping off the lotion on the nipples before feeding the baby may d. Wiping off the lotion on the nipples before feeding the baby may help.help.

20

4.4. The alternatives should sample the same domain, for instance, all The alternatives should sample the same domain, for instance, all symptoms, all diagnostic tests, all nursing interventions, varying symptoms, all diagnostic tests, all nursing interventions, varying treatments, and so forth.treatments, and so forth.

You are examining a patient who is having difficulty breathing. HisYou are examining a patient who is having difficulty breathing. His

RR is 40, HR 140, and oxygen saturation 90%. He also complains ofRR is 40, HR 140, and oxygen saturation 90%. He also complains of

headache.headache.

Which of the following nursing diagnoses is of greatest priority?Which of the following nursing diagnoses is of greatest priority?

a. Activity intolerancea. Activity intolerance

b. COPDb. COPD

c. Impaired gas exchange*c. Impaired gas exchange*

d. Paind. Pain

21

5.5. Avoid including opposite responses among the options. This is Avoid including opposite responses among the options. This is often a clue to choose between the opposites and not consider the often a clue to choose between the opposites and not consider the others. A sample item follows:others. A sample item follows:

The nurse should determine the correct placement of a nasogastricThe nurse should determine the correct placement of a nasogastric

tube by:tube by:

a. asking the patient to swallow.a. asking the patient to swallow.

b. aspirating gastric fluid from the tube.b. aspirating gastric fluid from the tube.

c. inserting air in the tube and auscultating in the epigastric area.*c. inserting air in the tube and auscultating in the epigastric area.*

d. inserting water in the tube and auscultating in the epigastric area.d. inserting water in the tube and auscultating in the epigastric area.

In this example, the correct response (c) is opposite one of theIn this example, the correct response (c) is opposite one of the

distractors (b), which clues the student to select one of thesedistractors (b), which clues the student to select one of these

alternatives. In addition, options “c” and “d” begin with “inserting,”alternatives. In addition, options “c” and “d” begin with “inserting,”

which may provide a visual clue to choose between them.which may provide a visual clue to choose between them.

22

McDonald (2007) suggested that when two sets of opposites are usedMcDonald (2007) suggested that when two sets of opposites are usedin the alternatives, there is less opportunity for guessing. Using thisin the alternatives, there is less opportunity for guessing. Using thisprinciple, the first distractor in the example could be reworded to principle, the first distractor in the example could be reworded to

formforma second pair of opposites:a second pair of opposites:The nurse should determine the correct placement of a nasogastricThe nurse should determine the correct placement of a nasogastrictube by:tube by:a. aspirating air from the tube.a. aspirating air from the tube.b. aspirating gastric fluid from the tube.b. aspirating gastric fluid from the tube.c. inserting air in the tube and auscultating in the epigastric area.*c. inserting air in the tube and auscultating in the epigastric area.*d. inserting water in the tube and auscultating in the epigastric area.d. inserting water in the tube and auscultating in the epigastric area.

23

6.6. Arrange the options in alphabetical, numerical, or chronological Arrange the options in alphabetical, numerical, or chronological order, so that the position of the correct response is randomly order, so that the position of the correct response is randomly distributed throughout the test.distributed throughout the test.

7.7. Options with numerical values should be listed sequentially, and Options with numerical values should be listed sequentially, and the values should not overlap. When alternatives overlap, a the values should not overlap. When alternatives overlap, a portion of anportion of an option may be correct, or more than one answer option may be correct, or more than one answer may be possible. An example of this problem is:may be possible. An example of this problem is:

The normal range for serum potassium level in adults is:The normal range for serum potassium level in adults is:a. 2.5 – 4.5 mEq/L.a. 2.5 – 4.5 mEq/L.b. 0.5 – 3.5 mEq/L.b. 0.5 – 3.5 mEq/L.c. 3.5 – 5.2 mEq/L.*c. 3.5 – 5.2 mEq/L.*d. 1.5 – 4.5 mEq/L.d. 1.5 – 4.5 mEq/L.To correct this problem, see next slide.To correct this problem, see next slide.

24

The normal range for serum potassium level in adults is:The normal range for serum potassium level in adults is:

• a. 0.5 – 1.5 mEq/L.a. 0.5 – 1.5 mEq/L.

• b. 2.0 – 3.2 mEq/L.b. 2.0 – 3.2 mEq/L.

• c. 3.5 – 5.2 mEq/L.*c. 3.5 – 5.2 mEq/L.*

• d. 8.5 – 10.3 mEq/L.d. 8.5 – 10.3 mEq/L.

2525

Multiple-Response Items

In this format several alternatives may be correct. Students select all of the options that apply by checking the box that precedes each option, as in the following example:The preliminary diagnosis for your patient, a 20-year-old collegestudent, is meningitis. Which signs and symptoms should you anticipate finding? Select all that apply:

❒ 1. Abdominal tenderness ❒ 2. Fever * ❒ 3. Lack of pain with sudden head movements ❒ 4. Nausea and vomiting * ❒ 5. Nuchal rigidity * ❒ 6. Sensitivity to light * ❒ 7. Sudden bruising in neck area

26

The principles for writing multiple-response items are the same as for writing multiple-choice. An additional suggestion for writing these items is:The responses should be listed in a logical order, for instance,alphabetically, for ease in reviewing. Alternatives are easier to review if shorter combinations are listed before longer ones. For example, Causes of cataracts include:1. aging.2. arteriosclerosis.3. hemorrhage.4. iritis.5. steroid therapy.a. 1, 2b. 1, 5*c. 2, 4d. 1, 3, 4e. 2, 3, 5

Conclusions

• Because testing is a core component of the educational process, many universities offer training to faculty members involved in the process of writing tests.

• Nurse educators and others at the UoH are responsible for the quality of the tests they develop. Poorly written test items do not meet the main aim for which they were generated; they are meaningless, have no value, and a waste of time and effort. Added to this the fact that the results such test yield are by no means reliable and truly reflect students’ mastery of a taught content.

• Pre-requisites for successful item writing include proper command of English, knowledge of principles and rules of item construction, and sufficient time.

• Studies investigating the quality of teacher-constructed test items need to be carried out as soon as possible.

A Final Question

• Besides adhering to the principles and rules of item construction, how would you know that your test is of good quality?

Where to go from here

• Access my Homepage at Access my Homepage at http://faculty.uoh.edu.sa/b.hijah/ and download the seminar. You may wish to find out whether you have successfully applied the said rules and principle to your previous MC tests.

• Make use of the information presented in this seminar when Make use of the information presented in this seminar when you develop your tests in the future. you develop your tests in the future.

303030

References • Downing, S.M. (2005). The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education 10 (2), 133–143.

• Ellsworth, R.A., Dunnell, P., Duell, O.K. (1990). Multiple-choice test items: what are textbook authors telling teachers? Journal of Educational Research 83 (5), 289–293.

• Hansen, J.D. (1997). Quality multiple-choice test questions: Item writing guidelines and an analysis of auditing test banks. Journal of Education for Business 73 (2), 94–97.

• Jozefowicz, R.F., Koeppen, B.M., Case, S., Galbraith, R., Swanson, D., Glew, R.H. (2002). The quality of in-house medical school examinations. Academic

Medicine 77 (2), 156–161.

313131

References (Continued…)

• Masters, J.; Hulsmeyer, B.; Pike, M.; Leichty, K.; Miller, M.; & Verst, A. (2001). Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education. Journal of Nursing Education. 40, 25–32. • McDonald, M. E. (2007). The nurse educator’s guide to assessing learning outcomes. Boston: Jones and Bartlett.

• Oermann, M. & Gaberson, K. (2008). Evaluation and Testing in Nursing Education. 3rd edition. New York: Springer.

• Shizuka T, Takeuchi O, Yashima T, Yoshizawa K (2006). A comparison of three- and four option English tests for university entrance selection purposes in Japan. Language Testing 2006;23:35–57.

• Tarrant, M.; Knierim, A.; Hayes, S.; and Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today. 26, 662–671.

323232

References (Continued…)

• Tarrant, M.; Ware, J. and Mohammed, A. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education. 9:40 doi:10.1186/1472-6920-9-40.

33

Thank You Very Much

Any Questions?