1

Multiple Choice Questions(MCQs)

Ensuring the question fits the purpose

2

Your aims for the session

• Please discuss where you are at with the use of MCQs and what you are aiming to achieve by the end of this session with the person next to you

3

When setting up this workshop our aims were to cover:

• An overview of assessment principles • How to develop basic multiple choice

questions• How to use MCQs in the assessment process

I.e. some theory and some hands on….

And from your aims…..

4

Use of MCQs at UH• Paper

– In class tests– Written exams

• StudyNet quizzes

• EVS– Within sessions

• Check understanding• Feedback for staff

– Assessment• Formative• Summative

– Homework function

• QMP (Questionmark Perception)

5

Considerations when creating MCQs Pedagogical

UH Assessment for Learning Principles

• Engages students with the assessment criteria

• Supports personalised learning• Ensures feedback leads to

improvement• Focuses on student

development• Stimulates dialogue• Considers student and staff

effort

Good practice in Undergraduate Education

• encourages contact between students and lecturers

• develops reciprocity and cooperation among students

• encourages active learning • gives prompt feedback • emphasises time on task • communicates high expectations • respects diverse talents

(Chickering and Gamson,1987)

6

Considerations when creating MCQsPedagogical

1) Learning outcomes Module

Session

– SEEC level descriptors

2) Considering the effectiveness and efficiency of practice and assessment from both students’ and staffs’ viewpoint

7

Considerations when creating MCQsEffective vs. efficient assessment

Waste of time

Educational Effectiveness

Sta

ff/R

esou

rce

Effi

cien

cy (o

r 1/t)

8

Considerations when creating MCQsBloom’s most quoted domain – the cognitive

9

Revised by Anderson in the 1990’sNouns to verbs and the top two reversed

10

Considerations when creating MCQsSEEC level descriptorsSouthern England Consortium for Credit Accumulation and Transfer

• Level 3-8– Setting– Knowledge and understanding– Cognitive skills– Performance and practice– Personal and enabling skills

• Task for you….On your tables you have the SEEC level descriptors for 4,5+6 …..without the level headings….which is which….??

11

Thus far..Module

LOs

Session LOs

SEEC level

descriptors

Bloom’s /Anders

on

MCQ

12

Types of MCQs• Simple

– True/False– One correct answer

• Multiple Response Questions– Multiple True/False– Matching items– Extended matching– Assertion Reason

– Marking gets more complex• E.g. Confidence marking/Negative marking

13

Some examples

• To start you thinking

• Locate your EVS clicker…….

The main stadium for the London Olympics in 2012 is located in:

14

1. Great Britain2. British Isles3. England4. Scotland

15

Quintin McKellar who was previously VC at the Vet School joined the University in?

1. December 20102. January 20113. February 20114. January 2020

16

MCQs

1. Are reliable2. Are easy to mark3. Are easy to produce4. Can only be used to

test the cognitive domain

17

Possible problems around those questions?

18

Components of an MCQ

Good MCQ

Well formed STEM

Clearly correct answer

Plausible distractor

s

Options

+/- feedback

19

Stems • Considerations

– Use clear, straight forward language– Aim to write as a complete sentence– Avoid use of unnecessary content– Avoid the use of negatives. If you must use them

embolden them to make them stand out– Avoid giving clues in the question e.g an/a– Avoid veering away from the content/cognitive

level that you aimed to address

Options

Correct answer and Distractors

– Distractors should be worded in a similar way to the correct answer and of a similar length (but not too close to cause confusion)

– Avoid repeating the same content in the options – move it to the stem if possible

– Avoid ‘all of the above’/ ‘none of the above– Aim to use common misconceptions/student errors as distractors– Distractors should be plausible– There should be a sufficient number of distractors.

• A correct statement but not the answer to the question is a good distractor

20

21

On each table – using the hand-outCan you write better stems and options for the

previous 3 questions?

The main stadium for the London Olympics in 2012 is located in?

Quintin McKellar who was previously VC at the Vet School joined the University in?

MCQs…..

22

True/FalseTwo possible alternatives so fair chance of getting the right answer…

• Some tips:– As in multiple choice questions generally, use negatives sparingly.– Use statements which are unequivocally true or false.– Avoid lifting statements directly from assigned reading, lecture notes or other

course materials so that recall alone will not permit a correct answer.– Generally avoid the use of words which would signal the correct response to

the test-wise student. Absolutes such as “none”, “never”, “always”, “all”, “impossible” tend to be false, while qualifiers such as “usually”, “generally”, “sometimes” “often” are likely to be true.

• More usually used as a basis for more complex Assertion/Reason questions

23

Any questions on basic MCQ writing at lower levels of Bloom/SEEC

24

Higher level of cognitive testing

• Multiple Response Questions– Multiple True/False– Matching items– Extended matching– Assertion Reason

• Marking gets more complex• Some forms can be used in EVS but not all

– E.g. Confidence marking/Negative marking

• Students rate their certainty on a scale of 1-3

• CBM motivates reflection about the reliability and justification for each answer, since the student gains by identifying reasons either for reservation or confidence

• Maximum formative benefit requires immediate feedback, prompting reflection about uncertainties or confident errors

Certainty-Based Marking

Why use CBM? • To encourage them to really understand the issues, not just

to react immediately to a question• To encourage students to think laterally: other pieces of

knowledge may help to validate or question their answer• To challenge the students - if they won't risk losing marks if

wrong, then they don't really know the answer.• It will encourage the students to have confidence in their

knowledgeIt is more fair - a thoughtful and confident correct answer deserves more marks than a lucky hunch

• Efficient study requires that students constantly question how their ideas arise, and how reliable they are

• It encourages a reflective approach to learning

Gardner-Medwin, 2006

27

Multiple true falseStudent’s clinical judgement is assessed:

A 28 year old woman with one child has taken anti-thyroid drugs for 6 months for thyrotoxicosis. She has a friend who has been successfully treated with radioiodine. She finds she frequently forgets to take her drugs and wants to stop them to have radio-iodine treatment.

A. She should be told that because of her age radio-iodine is best avoided.B. The problems associated with radio-iodine should be discussed with her.C. Surgery as a possible alternative should be discussed with her.D. She should be advised that some form of further treatment is required.E. You should find out more about her friend’s treatment.

Correct answer: - true, B, C and D: - false, A and E.

Examples from Harden, R.M. & Dunn, W.G. (1981) Assessment a Work ManualDundee Centre of Medical Education in Brown et al 1997 from Loughborough University

28

More complex MCQ – termed matching

Simple MCQ

Acute intermittent porphyria is a result of a defect in the biosynthetic pathway for:

A. collagenB. corticosteroidC. fatty acidD. glucose E. haem*F. thyroxine (T4)

Matching MCQ

An otherwise healthy 33-year-old man has mild weakness and occasional episodes of steady, severe abdominal pain, but no diarrhoea. One aunt and a cousin have had similar episodes. During an episode his abdomen is distended, and bowel sounds are decreased. Neurological examination shows a mild weakness in the upper arms. These finding suggest a defect in the biosynthetic pathway for:

A. collagenB. corticosteroidC. fatty acidD. glucose E. haem*F. thyroxine (T4)

Versus

29

Extended matching

• More complex with multiple questions and answers on a common theme

• Basis– A theme – Lead in statement– Multiple scenarios/questions – Options

– An example - next 2 slides

30

Extended MatchingMultiple questions and multiple answers

For each of the following patients with a myocardial infarction select the most likely complication from the list below. Each option may be used once, more than once or not at all. A) A 72 year old man is admitted to the Coronary Care Unit following a myocardial

infarction. Seven days later he complains of severe trouble breathing and soon after, he collapses. On examination he is pale. His right leg appears swollen compared to the left. The apex beat is at the 5th intercostal space, mid-clavicular line. There are no murmurs audible but there are occasional bi-basal crackles in his chest. His pulse is 128/min, blood pressure 98/55 mmHg, temperature 37.7 C and jugular venous pressure is elevated.

B) A 62 year old woman is admitted to the Coronary Care Unit following a myocardial infarction. Three days later she has become very unwell. On examination she is breathing noisily and her ankles are swollen. The apex beat is at the 6th intercostal space, anterior axillary line. There is a pansystolic murmur radiating to the axilla. Examination of the lungs reveals crackles at both bases and midzones. Her pulse is 126/min, blood pressure 105/65 mmHg, temperature 37.3 C, and respiratory rate 24/min. Her jugular venous pressure is raised etc..

31

• Asystole• Atrial fibrillation• Cardiac tamponnade• Complete heart block• Deep vein thrombosis• Papilliary muscle rupture• Pericarditis• Post myocardial infarction syndrome• Pulmonary embolism• Pulmonary oedema• Unstable angina• Ventricular fibrillation• Ventricular septal rupture• Ventricular tachycardia

Options

From the Medical Schools Council, 2012

32

Assertion Reason (ARQs)• Assertion-reason questions are used to

explore cause and effect and identify relationships between statements and hence test application/analysis.

• When writing assertion-reason questions, keep in mind the following points: The reason should be a free standing

sentence so that it can be considered separately from the assertion.

Avoid using minor reasons. These can result in an ambiguous question.

Repeat options A-E in full for each question.

33

Example of assertion reasonA True/True

Reason is correct explanation

B True/TrueReason Is NOT a correct

explanationC True/FalseD False/TrueE False/False

Assertion High speed is a factor in car accidents

Reason Most modern cars can reach speeds in excess of 100mph

BECAUSE

34

Activity• For each table there is an example of a type of

a more complex Multiple Response Question• In your group devise a question using

principles from last task• At the end pass your question to the next

table for feedback…. Asking for:– is the stem clearly worded ?– are the options clear and unambiguous?– what level of Bloom/Anderson does it address?

35

Feedback from Activity

36

Can also use ‘simple’ MCQs to test higher cognitive levels

• Application/Analysis and Evaluation

37

MCQ for testing Analysis• "The story is told of the famous German Organic Chemist Auguste Kékulé who

was struggling with the problem of how the six carbon atoms of benzene were linked together. He was getting nowhere with the problem, and one day fell asleep in front of the fireplace while he was pondering on it. He dreamt of molecules twisting and turning around like snakes. Suddenly, one of the snakes swallowed its own tail and rolled around like a hoop. Kékulé woke up with a start, and realized that his problem could be solved if the six carbon atoms of benzene were attached to each other to form a ring. Further work showed that this was entirely correct."

The above passage illustrates a particular phase of the creative process. Which one is it?1. preparation 2. incubation 3. orientation 4. illumination 5. verification

38

MCQ for testing EvaluationQuestion – define the creative process“The creative process is believed to take place in five stages, in the following order: ORIENTATION, when the problem must be identified and defined, PREPARATION, when all the possible information about the problem is collected, INCUBATION, when there is a period where no solution seems in sight and the person is often busy with other tasks, ILLUMINATION, when the person experiences a general idea of how to arrive at a solution to the problem, and finally VERIFICATION, when the person determines whether the solution is the right one for the problem."

How would you judge this student' s answer?1. EXCELLENT (all stages correct in the right order with clear and correct explanations) 2. GOOD (all stages correct in the right order, but the explanations are not as clear as they should be). 3. MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR the explanations are not clear OR the explanations are irrelevant) 4. UNACCEPTABLE (more than two stages are missing AND the order is incorrect AND the explanations are not clear AND/OR they are irrelevant)

39

Checklist for questions

Do all the questions?• Test the level that was

intended• Have a clear stem• Have only one clearly

correct answer • Have a distractor that is

similar in wording and length to the correct one

• Have plausible distractors

Do the questions avoid?• Repeating text in the

question and the answer • Double negatives• Use of ‘all of the above’/

‘none of the above’• Giving clues in the question

(e.g. use of an/a etc)• Cultural etc bias

40

Checklist for questions as a whole

• Does the test overall:– Test the range of content?– Test the range of:

• knowledge/comprehension/application/analysis/ evaluation/synthesis that was intended?

– Avoid giving away answers to questions in other questions?

– Have a range of a/b/c/etc correct answers– Consider inclusivity

41

Pros and Cons of MCQs generallyParticipant's thoughts ?

Advantages• Banks of questions created

and can be used for reuse/sharing

• Reliable (if set correctly…)• Marking is time efficient, no

second marking required• Rapid feedback to students

and staff • Enables assessment of large

range of curriculum

Disadvantages• Time to compile questions• Skill and scrutiny still required in the

setting process• Unable to test practical skills• Difficulties in compiling questions for

higher order cognitive skills cannot assess innovation/creativity

• Concerns re guessing • Partial knowledge not recognised in

more straight forward questions• If only used formatively may give

students unreal expectations in other forms of assessment

• Concerns re relatively high marks

From the workshop flyer the objectives were……..

By the end of the session the participants will:• Be conversant with the assessment principles• Be able to write basic level MCQs• Identify how MCQs may be used in the

assessment process

• Have we achieved these? if not plans to fulfil them….

42

43

References and BibliographyAnderson, L . W., Krathwohl D. R., Airasian, P. W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., James Raths, & Wittrock, M.C. (eds). (2000). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Allyn and Bacon

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: the classification of educational goals; Handbook I: Cognitive Domain New York, Longmans, Green, 1956

Brown, G., with Bull, J., and Pendlebury, M. (1997). Assessing Student Learning in HigherEducation. London: Routledge.

Chickering, A.W. & Gamson, Z.F. (1987). Seven Principles for good Practice in Undergraduate Education. American Association for Higher Education Bulletin, 39(7): 3-7. Accessed 2/6/12

Designing and Managing MCQs University of Cape Town

Gardner-Medwin AR (2006) Confidence-Based Marking – towards deeper learning and better exams In : Innovative Assessment in Higher Education. Ed.: Bryan C and Clegg K. Routledge, Taylor and Francis Group, London

Fry, H., Ketteridge. S. and Marshall, S. (2005) A handbook for teaching and learning in higher education. Enhancing academic practice. 2nd edition. Routledge Falmer . Oxon.

HEA Legal Centre – how can I write effective MCQs? University of Leeds

44

References and BibliographyHornby, Win (2003) Dogs, Stars, Rolls Royces and Old Double Decker Buses: Efficiency and effectiveness in assessment. Accessed on-line 24/6/12 http://www.enhancementthemes.ac.uk/docs/publications/reflections-on-assessment-volume-i.pdf

Quinn F.M & Hughes S (2007) Quinn’s principles and practice in nurse education. 5th edition. Cengage learning. UK.

A Guide to using Multiple Choice Questions and other Objective Test Questions La Trobe University

Writing multiple choice questions Virginia Commonwealth University

Constructing Written Test Questions For the Basic and Clinical Sciences (2002). 3rd Ed. National Board of Medical Examiners

Designing effective objective test questions: an introductory workshop - Computer Assisted Assessment - Loughborough University

Writing MCQs for clickers is different (Pb wiki) - part of a whole section on writing MCQs

SEEC Credit Level Descriptors for Higher Education (2010) http://www.seec.org.uk/sites/seec.org.uk/files/SEEC%20Level%20Descriptors%202010_0.pdf

Numerous medical books with examples of MCQs used for the medical Membership exams (MRCP)