Upload
liz-norman
View
72
Download
1
Embed Size (px)
Citation preview
Writing short and long answer
exam questions
Liz Norman
Massey University
“Effective item writers are trained, not born” (Downing, 2006)
Steven M Downing (2006) Twelve steps for effective test development, In: Downing SM and Haladyna TM (Eds), Handbook of Test Development, Lawrence Erlbaum Associates, Mahwah, NJ, USA, p.11
Breadth and level of the questions
• Are the questions focused on middle to higher level tasks such as creation, evaluation, analysis and application?
Recall-higher order classification
Fact recall Questions capable of being answered by
reference to one paragraph in a text or notes
(or several paragraphs for questions requiring
recall of several facts)
Applied
(higher order)
Questions that require the use of facts or
concepts, the solution of a diagnostic or
physiologic problem, the perception of a
relationship, or other process beyond recalling
discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.
Knowledge-understanding classification
Knowledge knowing about; a body of coherent facts; can be
thought of as right or wrong.
Understanding knowing how and why; the meaning of facts; the
theory that links facts and provides meaning; how
sense is made of facts to enable them to be applied
to analysis, synthesis, evaluation; to be able to
explain why particular facts or skills are applicable
to a particular situation; to know which fact to
apply when; to be able to create new knowledge or
modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.
SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
Prestructural Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
Quantitative change
Qualitative change
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
surface
deep
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
recall
application
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
knowledge
understanding
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise, find, note,
seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do
algorithm, apply method, account for execute, formulate, solve,
conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes,
apply theory (to its own domain), argue, implement, plan,
summarize, construct, design, interpret (some senses), structure,
conclude, substantiate, exemplify, derive, adapt
Extended
abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory
(to new domain), assess, evaluate, interpret (some senses),
critically reflect, predict, criticise, reason
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
Webb’s depth of knowledge
• Level 1 – recall– Recall information or perform a simple step, identify, measure,
describe, explain simple ideas
• Level 2 – skill/concept– Requires more than one step, comparing, interpreting,
estimating, making observations, explaining, organising and displaying data
• Level 3 – strategic thinking– Requires planning and using evidence, explaining reasoning,
make conjectures, draw conclusions, solving problems
• Level 4 – extended thinking– Complex reasoning, planning developing and thinking over an
extended period of time, making multiple connections, synthesis of ideas into new concepts
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25
Breadth and level of the questions
• Are the questions focused on middle to higher level tasks such as creation, evaluation, analysis and application?
• Are the questions at the right level of difficulty?
Are the questions at the right level of difficulty?
• There needs to be a match between the depth of knowledge and degree of skill (both practical and cognitive) required to answer the question, and those specified for the topic in the learning outcomes.
• Learning outcomes specify what students are able to do when they have completed the course—so you need to think about the level of study of the student
• Day one competencies
Some factors that affect demand and difficulty
• Type of operation
• Degree of novelty
• Number of components or ideas involved
• Whether resources are provided or need to be generated by the candidate
• The question wording and any images, diagrams, or tables provided
• Degree of abstraction
• Response strategy – simple, stepwise, integrated
For EACH ECG, make a recommendation for management of the horse and explain your reasoning.
Question 1:
Explain the physiological actions of insulin.
Question 2:
Explain the physiological actions of ghrelin.
Question 1:
Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.
Question 2:
Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
Question 3:
Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.
Question 4:
Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.
Describe the skull. (30 marks)
Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)
Question 2a) Describe the advantages and disadvantages of insulin
therapy for diabetes mellitus in cats (10 marks)b) Indicate the dose and frequency of administration of
insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).
You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.
Breadth and level of the questions
• Are the questions focused on middle to higher level tasks such as creation, evaluation, analysis and application?
• Are the questions at the right level of difficulty?
• Do the questions provide the right breadth of coverage and relative weighting of the learning outcomes?
Example weighting system
Scores for impact and frequency are multiplied to give a total weighting for each curriculum area.
McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547.
Impact Weight Frequency Weight
Less important 1 Rarely seen 1
Essential 2 Relatively common 2
High impact 3 Very common 3
Examination structure
• Do the examination materials follow the exam structure format as described in the paper details?
• Are marks allocated to each question and separate subpart?
• Are the marks allocated to each question and each sub-part appropriate?
• Do the marks add up to the expected total?
• Is there sufficient time to answer all the questions?
Pacing examinations
• Reading time: 40 words per minute (Klatt & Klatt,
2011).
• Writing time: 16-20 words per minute (Summers
& Catarro, 2003)
• Marking schemes help you check the pacing
• Set exam to be 1 mark = 1 minute
Klatt, E. C., & Klatt, C. A. (2011). How much is too much reading for medical students? Assigned reading and reading rates at one medical school. Academic Medicine, 86(9), 1079-1083. Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157
Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157
Question structure
• Is it clear for each question what is required in terms of scope?
• Does each question contain an instructional verb that explains the task required to the student?
Don’t write questions; write tasks
What is your diagnosis?
State the most likely diagnosis
State the most likely diagnosis and explain your reasoning
Discuss the differential diagnoses you would consider in this case
or …..
Instructional verb examples
Compare: to find similarities between things, or to look for characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.
Question structure
• Is it clear for each question what is required in terms of scope?
• Does each question contain an instructional verb that explains the task required to the student? Questions should not really be questions: they should be instructions instead.
Make short notes on THREE (3) of the following.
a) Pulmonary compliance.
b) Bicarbonate as a blood buffer.
c) The respiratory centre.
d) The respiratory functions of the nose.
Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.
A dog is presented to your clinic after a road traffic accident.
a) What clinical signs would be consistent with urinary bladder rupture?
Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3
Question structure
• Is it clear for each question what is required in terms of scope?
• Does each question contain an instructional verb that explains the task required to the student? Questions should not really be questions: they should be instructions instead.
• Does each question contain ONLY relevant content and no irrelevant content?
A client rings you up to arrange his quarterly visit as he is frustrated by ongoingproblems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.
On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.
You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)
Which of the following statements is TRUE?
a. A ventral hernia is classified as a false hernia
b. A diaphragmatic hernia is classified as a true hernia
c. Umbilical hernias are classified as false hernias
d. False hernias have a hernia sac
e. Umbilical hernias are relatively uncommon
Question structure
• Is it clear for each question what is required in terms of scope?
• Does each question contain an instructional verb that explains the task required to the student? Questions should not really be questions: they should be instructions instead.
• Does each question contain ONLY relevant content and no irrelevant content?
Question structure continued
• Are any ancillary materials used (images, radiographs, histopathology slides etc) fit for purpose, clear, legible and of good quality?
• Is a marking scheme provided which indicates the agreed standard by which students’ answers will be judged and marks awarded?
• Does the answer provided in the marking scheme answer the question that is being asked?
• Do you agree with the marking scheme?
Grammar and spelling
• Is the language used in each question clear and grammatically correct?
• Are there any spelling mistakes?
• Are units given as SI units, or the units that the student has been using in learning activities?
• Are any numerical values used realistic and correct?
http://www.slideshare.net/liznorman