Gordon Page Emeritus Professor, Medicine University of

Gordon PageEmeritus Professor, MedicineUniversity of British Columbia

Vancouver Canada

British ColumbiaBritish Columbia

* Prince George

* VancouverVictoria *

* Okanagan

PBL

Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learningto new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments


“Assessment for Improved Transfer and Long Term Outcome”

Transfer: the skill of applying what was learned to new situations (Mayer, 2004)*

In medical education, there is:Short-term transfer – applying what was learned earlier to what is currently being learned, and Long-term transfer – applying what was learned to the ‘real world’ care of patients (‘Long term outcome’)

*Mayer, Richard E., “Should There Be a Three‐Strikes Rule Against Pure Discovery Learning? “, American Psychologist, Vol 59(1), Jan 2004, 14‐19.

But, ‘transfer’ has an additional meaning in the context of assessment …The ‘transfer’ of assessment results to new situations – the degree to which assessment results are indicators of students’ performance in new situations.New situations include further training, and ultimately the independent care of patients.

So …, there are two aspects to ‘transfer’ in the context of assessment

Transfer of learning to new situationsTransfer of assessment results to new situations

To address the title of this presentation, our challenge as medical educators is …

How can we design our assessments to:

better assist students to transfer their previous learning to new situations?enable more accurate inferences about the future performance of students in new situations

Purpose of the PresentationExplain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learning to new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments

The Challenge

How can we design our assessment strategies to enhance students’ short and long term transfer of learning?

What conditions must be met to enable transfer of learning?

The student must be able to recall the knowledge or skill to be transferred.The student must understand the knowledge or be proficient in the skill to be transferredThe student must see the relevance of the knowledge or skill to the new situation

How do we design assessments to enhance recall, understanding/proficiency, and

recognition of relevance?

Key instructional strategies used to enhance the transfer of learning …

Learning in context (e.g., via integrated systems-based curricula, case/problem based curricula, learning while caring for patients)Practice in using the knowledge/skillFeedback on performance

But what are the key assessment strategies to enhance transfer of learning?

“Back to Basics”

How can transfer of learning be effectively addressed in our assessment strategies?

1. Take advantage of the adage, ‘assessment drives learning’

Assessment is the most powerful tool we possess for conveying to students

what is important to learn. Our challenge therefore is to test what is

important to learn!

Utility (U) of an Assessment Strategy

R = ReliabilityV = Validity(E = Educational impact)A = AcceptabilityC = Cost

U = R x V x E x A x C


2. Test students’ ability to apply their knowledge in contrast to recalling their knowledge

Test ‘in context’

Why test ‘application’?

Clinical Reasoning and Decision-Making

“How do clinicians use their knowledge to arrive at diagnostic and management decisions?”


(Bordage, G. Academic Medicine, 1994, 1999)

Reduced DispersedElaborated Compiled


(Bordage, G. Academic Medicine, 1994, 1999)

Reduced (“empty mind”)Dispersed (“cluttered mind”)Elaborated (“deductive thinking”)Compiled (“recall/recognition”)

Factual RecallWhat is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

(A) acute poststreptococcal glomerulonephritis

(B) hemolytic-uremic syndrome*(C) minimal change nephrotic syndrome(D) nephrotic syndrome due to focal and

segmental glomerulosclerosis(E) Schönlein-Henoch purpura with nephritis

ApplicationA 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, pp 58-9.

What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

A B *C* D E Overall P-value

1 0 99 0 08 1 90 1 0 94

A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. What is the most likely diagnosis?

0 1 98 1 010 9 66 10 5 84

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58‐9.


ReducedDispersed

ElaboratedCompiled


3. By providing students with feedback on their performance

Feedback is most effective if it is timely (immediate, a ‘teachable moment’), specific, descriptive, honest, and

challenges the student to self-assess their own performance

Summary -- transfer of learning can be effectively addressed in our assessment strategies by …

1. By taking advantage of the adage, ‘assessment drives learning’

2. By testing students’ ability to apply their knowledge in contrast to recalling their knowledge

3. By providing students with feedback on their performance

Context Practice Feedback

Purpose of the Presentation

Explain the title, “Assessment for Improved Transfer and Long Term Outcome”Identify assessment strategies and guiding principles for enhancing transfer of learning to new situationsIdentify assessment strategies and guiding principles for enhancing transfer of assessment results to new situations Provide a ‘report card’ on the effectiveness of current assessment practices in contributing to transfer and long term outcomesIdentify where we should focus our future efforts to achieve better transfer and long term outcomes from our assessments

The Question

Which assessment strategies enhance transfer of assessment

results to new situations?

Transfer: The Missing Link inMedical Education

Geoff Norman, Ph.D.McMaster University

[email protected] 2010

Transfer and Assessment

Every test is a simulation of reality

Multiple choice (MCQ)

Key Features Problems

OSCE

Human Patient Simulator (HPS)

The move from performance on simulation to performance in reality = Transfer

Authenticity/fidelity

Assumptions underlying Test Scores

Score on a test is correlated with a “Score” on reality

The more authentic (high fidelity) the test, the higher the correlation of the test score with a ‘score’ on reality

Is there evidence supporting the validity of these assumptions?

David Swanson, NBMEMay 2010

Does greater fidelity = Greater transfer?

“How much authenticity is enough?”

“It is possible to have too much authenticity”

The Ultimate Simulator ‐‐747

Meet “HARVEY”

“Harvey” combines a life-sized manikin with JV, carotoid and peripheral pulses and realistic heart sounds with computer modeling of cardio physiology… able to produce realistic symptom complexes.”

Hi Harvey!

Geoff Norman’s meta analysis summary of findings from studies of transfer of learning from hi fidelity simulators

Hi fi simulators show small, non-significant learning gains over lo-fidelity simulation or didactic instruction

What about the transfer of assessment results to new

situations?

Geoff Norman: Relationship of Licensing Exam Scores to Complaints, Peer Assessments and Chart Reviews in Canada

Lower fidelity simulations (specifically key features examinations and MCQs) appear to provide better transfer than OSCEsRelation between assessment fidelity/authenticity and transfer of test results is questionable, and uncertain at bestThe missing elements interfering with transfer are not self-evident

??? Prototypical presentations ??? Noise (breath sounds, etc.)??? Anxiety, responsibility, (situated cognition)

Transfer from MCQ and OSCE Test Results to Performance in Clinical Practice (Mini-CEX)

Correlations between

Mini-CEX and MCQ scores .335Mini-CEX and OSCE scores .372

Meta analysis summary of the studies of transfer of assessment results to new situations

The transfer of assessment results to clinical practice situations is generally distressingly poor!Assessment results from low fidelity simulations appear to provide the best prediction of (tranfer to) future performance

ic

Rounds

Summary: Transfer of assessment results to new situations

In general, the transfer of assessment results to new situations is distressingly poor!Studies do not support the use of hi-fidelity simulations in improving transferThere is some evidence that assessments using lo fidelity simulation produce more ‘transferable’assessment results.


Enhancing Transfer and Long Term Outcomes -- Current Trends & Future

Directions

Assessment Strategy Current Trend

Future Direction

Test application of knowledge

Provide feedback on performance

Use of ICT/computers

Use of ‘lo fi’ simulation

Use of ‘hi-fi’ simulators

Observe students with patients

Review clinical performance,provide feedback


Directions


Future Direction









Directions


Future Direction









Directions


Future Direction



Use of ICT/computers ☺Use of ‘lo fi’ simulation





Directions


Future Direction



Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators




Directions

Assessment strategy Current Trend

Future Direction







Directions


Future Direction







Directions


Future Direction







Directions


Future Direction

Test application of knowledge ☺ ☺ ☺Provide feedback on performance





Directions


Future Direction

Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators




Directions


Future Direction

Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺Use of ‘hi-fi’ simulators




Directions


Future Direction

Test application of knowledge ☺ ☺ ☺Provide feedback on performance ☺ ☺ ☺Use of ICT/computers ☺ ☺ ☺Use of ‘lo fi’ simulation ☺ ☺ ☺ ☺Use of ‘hi-fi’ simulators




Directions


Future Direction





Directions


Future Direction


Observe students with patients ☺ ☺ ☺Review clinical performance,provide feedback


Directions


Future Direction


Observe students with patients ☺ ☺ ☺Review clinical performance,provide feedback

☺ ☺ ☺

“Back to Basics”

“Keep It Simple”

[email protected]

Danke SchÖn!

Documents

Gordon Page Emeritus Professor, Medicine University of