Liz Norman Examination and moderation guidelines

Examination and moderation guidelines

Liz NormanMassey University

Examination validity

• Exams are not valid or invalid – it’s the conclusions we draw from exam results that are valid or invalid.

• The entire conclusion about the correct grade for the student, can be called into question if any step is questionable.

• Just like in research, defensible conclusions require attention to methods and protocols.

6 principles to follow1. The content for examination must be appropriate2. The questions must elicit skills and knowledge that we want

to assess3. The questions must not elicit skills and knowledge that are

not part of the curriculum4. The mark scheme must reward the appropriate level of skill

and knowledge5. The questions must be the right level of difficulty6. The highest performing students should answer questions

well and the lowest performing students should answer them less well

1. The content for examination must be appropriate

• Content (as a whole) of all assessment for the paper should:– Reflect the weighting and importance in the curriculum– Involve skills and knowledge the students have practiced and

received feedback on• Do not reuse questions for more than 10-20% of the exam:

recall exams circulate even for total recovery exams• If using old questions, check the content is still relevant and

the material has been taught• Record procedures:

– blueprint – LOs covered by questions– where and when content was taught/practiced/fedback on

Breadth by learning outcome

Lit review assignment

Group presentation

Written examination

Practical examination

LO1 X X X

LO2 X X

LO3 X X X

LO4 X

Breadth across species

species number of Qs percent

small animal 15 52%

farm 5 17%

horse 4 14%

exotic 3 10%

lab 1 3%

all 1 3%

Breadth by topic

Pathophysiology Investigation and diagnosis

Treatment and management

Gastrointestinal P1Q1 P1Q1, P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

2. The questions must elicit skills and knowledge that we want to assess

• The wording of the questions needs to be explicitly clear, precise, and unambiguous

• Give instructions rather than asking questions• Use tasks that require higher order thinking• Images and other ancillary materials must be high

quality and clearly show what is intended• Moderators cannot check this without seeing the

mark scheme• Moderators from different disciplines may see issues

better than moderators from the same discipline

Don’t write questions; write tasks

What is your diagnosis?

State the most likely diagnosis

State the most likely diagnosis and explain your reasoning

Discuss the differential diagnoses you would consider in this case

or …..

Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

Contrast: to find differences or to distinguish between things.

Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

Illustrate: to use a picture, diagram or example to clarify a point.

Make short notes on THREE (3) of the following.a) Pulmonary compliance.b) Bicarbonate as a blood buffer.c) The respiratory centre.d) The respiratory functions of the nose.

Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.

Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.

A dog is presented to your clinic after a road traffic accident.a) What clinical signs would be consistent with urinary bladder rupture?

Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)

How would you localise the site of the lesion?

Answer provided in the marking scheme:Spinal lesion between T3 and L3

3. The questions must not elicit skills and knowledge that are not part of the curriculum• Common issues

– Timing issues (students can only write 16 wpm)– Reading speed– Vague questions (eg write short notes on…)– Content at the wrong level– Irrelevant content/images– Trivial/unimportant content

• Also remember– Colour blindness– Don’t use choice in examinations

• Sources of construct irrelevant difficulty are multiple and unpredictable: therefore – Keep question wording concise and straightforward– Only provide information required to answer the question and relevant distractors if

appropriate– Use images only when required to answer the question

Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157

Effect of time stress

• Time stress– Increases the use of schemas (sterotyping)– Decreases working memory processing capacity– Decreases the ability to maintain relevant

information and suppress irrelevant information

Question 1A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?

Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.


Question 2A ski pass costs $4.20. How much would it cost for 7 days?



Pass rate 41.5%Question 2A ski pass costs $4.20. How much would it cost for 7 days?

Pass rate 79%


Ahm

ed &

Pol

litt (2

007)

Impr

ovin

g th

e qu

ality

of c

onte

xtua

lized

que

stion

s: A

n ex

perim

enta

l in

vesti

gatio

n of

focu

s. A

sses

smen

t in

Educ

ation

: Prin

cipl

es, P

olic

y &

Pra

ctice

, 14(

2), 2

01-2

32.

4. The mark scheme must reward the appropriate level of skill and knowledge

• Avoid awarding marks for unprioritised volume of facts rather than relevant and structured knowledge

• Think about the quality of thinking

Fact recall vs applied

Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)

Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact

Peitzman et al. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.

Disconnected knowledge Connected knowledge

• No particular order to aspects presented

• Inclusion of irrelevant material• Inconsistencies or incorrect

aspects• Superficial or oversimplified• Replication of material from

sources – rote learned or reproduced without significant transformation

• Aspects explained relative to one another• Logically organised answer• Analysis and or synthesis• Compares similarities and differences• Integrates multiple levels (eg: molecular, biochemical,

systemic)• Evaluates inconsistencies• Expresses reasons• Explains implications or reaches a conclusion• Expresses relative importance, value, significance of

aspects. • Selective answer that addresses the point of the

question and may be shorter than a multistructural answer

• Uses the language of the discipline - terminology and phrasing

• Relates answer to examples or experience• Relates answer to organising principles of the discipline

Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.

4. The mark scheme must reward the appropriate level of skill and knowledge

• Avoid awarding marks for unprioritised volume of facts rather than relevant and structured knowledge

• Think about the quality of thinking• Holistic marking schemes are fine, but must specify

criteria to be used• Marking schemes may need refining during marking• Aiming for the median mark to be above 70%

5. The questions must be the right level of difficulty

• After/during marking check for overly difficult questions– Was there a problem with the question wording that has

meant students misinterpreted the intended task?– Was the answer keyed correctly (for MCQs)?– Was the content actually taught/practiced and given

feedback on?– Was there sufficient time to answer the question?

• May need to– Adjust marking scheme– Delete question – consult Liz or Jenny

6. The highest performing students should answer questions well; the lowest performing students should answer them less well

• After/during marking check all items with low or negative item to total score correlation. – Problems with question wording and interpreted task?– More than one right answer, exceptions that are also

correct?– Run out of time?– Even if you can’t see the problem, construct irrelevant

difficulty is likely• May need to

– Adjust marking scheme– Delete question – consult Liz or Jenny

http://www.slideshare.net/liznorman