Oral exams Liz Norman ANZCVS 2017

Oral exams

Liz NormanMassey University

insight reliability

Oral exams: strengths and limitations

insight reliability


• depth of knowledge• analytical and reasoning

capabilities• cognitive processes• decision making abilities• oral communication

insight reliability


• depth of knowledge• analytical and reasoning

capabilities• cognitive processes• decision making abilities• oral communication

• few cases/topics sampled• lack of examiner

agreement• assessing candidates on

different cases/topics• assessing different

numbers of cases/topics

insight reliability


• Ability to tailor questions and probing increases insight

• Long cases allow you to explore depth

• Standardisation of questions and probing increases reliability

• More short cases increases reliability

insight reliability


• Ability to tailor questions and probing increases insight

• Pre-prepared probes and marking schemes increases reliability

insight reliability


1. Finding the right balance2. Reducing examiner-examiner unreliability

Reducing examiner-examiner unreliability

• Examiner training– Use of exemplars of different standard– Practice in assessment

• Advance preparation of questions and prompts– Well-structured questions aimed at assessing appropriate skills– Clear and straightforward to answer

• Advance preparation of marking schemes– Common picture of performance levels: Familiarisation,

discussion, and agreement between examiners on the marking schemes

– Well structured marking schemes which are easy to apply and well-understood by examiners

Reducing examiner-examiner unreliability

• Allowing only preplanned questions and prompts– Planning for contingencies

• Putting candidates at ease as much as possible• Reducing examiner load as much as possible• Monitor examiner agreement• Consensus decision making

Finding the right balance: blueprinting

• Don’t aim to assess factual knowledge in oral exams – written exams are better for this

• Aim to assess:– Analytical, problem solving, reasoning skills– Cognitive processes– Decision-making abilities– Judgement– Oral communication– Professionalism, values, ethics (if these are part of

the subject guidelines)

Finding the right balance: questions

• Make your questions short, straightforward and clear

• Centre on clinical scenarios• Get straight to the point.

– Remove unnecessary information– Consider even saying what the point is:“This question is going to be about (disease x) and I want to explore your approach to management.”

• Keep Qs short - aim for new Q every 5 mins

Finding the right balance: questions

• Make Qs open-ended – Require interpretation, problem solving,

reasoning, judgement, or decisions– do not have one right answer

• Do not ask for personal experience or opinion– Eg: “How is this usually managed in your

practice?”, “What is your opinion on?”– Its “unmarkable” content– Can throw candidates. Differential effect on those

not “testwise”

Broad types of oral Qs

Recall: Questions that ask candidates only to recall facts, principles, or other information.

– List the most common causes of (x).– Define (x).– What is the most common disease associated

with (x)?– Explain two mechanisms by which (x) results

in (y).– Describe the differences between Type 1 and

Type 2 (x).

McGuire, C. H. (1966). The oral examination as a measure of professional competence. Journal of Medical Education, 41(3), 267-274.



Interpretive: Questions that require interpretation of information, and using principles to deal with new situations.

– Describe the changes on this radiograph.– Interpret the results of this biochemistry panel.– Describe the abnormalities you see (in this image).




Interpretive: Questions that require interpretation of information, and using principles to deal with new situations.

Problem solving: Questions that require finding a solution.– What is the list of differentials for these findings?– Outline you management of this case.– What would be the best next step in this case?– Can you account for the clinical findings in this case?


Finding the right balance: prompts, probes

• Plan them in advance but they can never be entirely standardised

• Only plan a few – need to move on to next case/topic

Types of prompts and probes

Exploring the depth of knowledge and higher order abilities

– Can you explain your reasoning?– What are the implications of each of these options?– What is the justification for your decision?– What evidence supports your judgement in this case?– Explain how you arrived at that conclusion– Because?– Why is that?– What would you do if your approach does not work?


Seeking more detail from the candidate– Any other differentials?– Can you tell me which specific drug you would

use?– Which particular suture pattern would be most

appropriate?– Which particular signs are leading you to conclude

that?– Can you elaborate on that?


Clarifying the question being asked or redirecting the candidate

– What I am getting at here is…– Can I get you to focus on (x) in this case?– Can you tell me all the options rather than what you would do?– Just coming back to something you mentioned earlier about (x),

can you tell me…

Asking about things that the candidate has missed out– Can you tell me any other implications? What about implications

for the whole farm?– One thing you didn’t mention was (x). Can you tell me about

that?

Discourse

Three different types of discourse in oral exams (Roberts, 2000)• Personal experience

– talk that deals with personal experience and feelings and often involves narratives and stories

• Professional– talk of professionals to clients and colleagues to

colleagues• Institutional

– principles: abstract, analytic, depersonalised.Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375

Discourse

• Important to frame your question in the same discourse that you expect an answer

• Usually you expect an answer in institutional discourse. – Candidates may not know this, especially those

inexperienced in this type of exam, from ethnic minorities or overseas trained.

– Causes candidate uncertainty and can result in differential bias in exam scores

• But in oral exams its easy to slip into one or other of the other discourses

Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375

The following question combines elements of both professional and personal experience.

Question: If someone took out a personal complaint against you, how would you react? Answer: That's where my personal stress management plan comes in . . . .

Although the theme of the question was professional and personal, the candidate responded in an institutional way. Instead of describing how she would feel (personal experience discourse) or what she did in her practice in relation to a specific case (professional discourse), she produced an abstract and analytical statement (institutional discourse) that satisfied the examiners as a response they could mark. Head nodding and mutual smiles provided confirmation.

Quote from Roberts, C. (2000). Oral examinations---equal opportunities, ethnicity, and fairness in the MRCGP commentary: Oral exams---get them right or don't bother. BMJ, 320(7231), 370-375

Ancillary materials

• Need to have clear function• Use them purposefully, not as decoration• Ensure they clearly show what they are

intended to show, when displayed in the way candidates will see them– Check with someone who did not see the case

Marking schemes

• Rate on only a few criteria eg– Factual recall– Analysis and interpretation of data– Problem solving– Verbal presentation

• Plus overall rating• Separately grade each main question/topic• Provide descriptors for 3-4 levels of performance

eg: clear fail, marginal fail, bare pass, good pass

Conducting oral exams

Create the right climate• Introductions and non-threatening questions (eg

weather)• Convey a sense of genuine interest• Adapt body language to candidate’s needs

– Some need eye contact, some find it off-putting– Candidate interpretations of “mhm” and “uhuh”:

• Encouragement to go on• Stop talking so I can ask the next Q• Well done you have passed

• Avoid humour

Not to much structure, not too little

• Plan all the questions and possible prompts including how these should be worded

• But on the day don’t read as if from a script• Adapt your language and body language to the

candidate• Change the order or leave out probes if necessary• Use clarification and redirection only if necessary• Give all candidates the same information• Metacommunicate

Dealing with issues

When the candidate can’t answer or says very little• Ensure candidate has had enough time to

respond (but don’t give them endless time)• Do not give hints• Ask the candidate if they would like the question

clarified. If they do, be careful not to– give hints or additional information– change the discourse or requirements of the answer

• Often best just to move on to next question

Dealing with issues

When candidates go off track or take too long• Use clarifying or redirecting questions

– What I am getting at here is…– Can I get you to focus on (x) in this case?– Can you tell me all the options rather than what

you would do?– Just coming back to something you mentioned

earlier about (x), can you tell me…

Dealing with issues

When candidates are exceedingly nervous• Trying to help often makes things worse• Poor performance may be the cause: move on

to the next question

Scoring candidates

• Record grade for each question as soon as it is finished• If you are thinking of giving quite a high grade think

“What would the candidate have to have done to earn an even higher grade?” – helps examiners use the whole scale

• At end of exam record your holistic judgement as well as sum/average of scores

• Do not confer with your co-examiners until you have recorded your own judgement

• Discuss your judgements and reach a consensus decision