Principles of feedback delivery and the role of self- assessment

Principles of feedback delivery and the

role of self-assessment

Kevin W. EvaDepartment of Clinical

Epidemiology and Biostatistics

Program for Educational Research and Development

McMaster University

Principles of feedback delivery

1. There are no universal principles

“There is no “best” type of formative feedback for all learners and all

learning outcomes”Shute (2008)

Click icon to add picture



2. Self-assessment is insufficient

The Rhetoric

• Almost every article on self-assessment begins with the same basic sentence:

“The ability to self-assess is a cornerstone of professional self-regulation”

The self-regulating professional

•Reflect regularly on daily practice

•Self-assess gaps in knowledge or skill

•Seek opportunities to redress gaps

•Invest energy to learn (or relearn)

•Incorporate knowledge into practice

•Repeat(Handfield-Jones, et al., 2002)

Eva, et al. (2004)

Before After Previous-0.2

0

0.2

0.4

0.6

0.8

1

Year 1Year 2Year 3

Corr

elat

ion

(r)

The self-assessment literature

• Hundreds of articles• Many literature reviews• One conclusion

Self-assessment ability

is generally poor

Boehler, Rogers, et al. (2006)

Pre-test Post-instruction Post-feedback0

5

10

15

20

25

Compliment group Feedback group

Perf

orm

ance

Rat

ing

“In a majority of relevant studies

physicians do not appear to

accurately self-assess”

Davis, et al. (2006)

The self-assessment literature

•Either the conclusions of hundreds of studies using multiple methodologies are all wrong or a critical premise underlying the concept of self-regulation in the professions is unsupportable

If you don’t think you’re susceptible, you’re not alone …

FAE Halo effect Cognitive dissonance

Reactive devaluation

12345678

Average Population Self Rating

Pronin, Lin, Ross (2002)

…

“Evolution deserves the Microsoft Windows Award for installing these mental processes in every one of us without asking permission.”

(Gilbert, 2006)



2. Self-assessment is insufficient

3. Self-assessment is vital

“It is impossible for a man to learn

what he thinks he already knows”

Epictetus (circa 100AD)


Quality of

feed-back

Eval of student

Self-evalua-

tion

Station order

Station The Student

00.050.1

0.150.2

0.250.3

0.35

Stan

dard

ized

beta

Predicting generation of learning goals

(Overall R2=.45; Eva, et al., In press)

“Physician agreement with

colleagues’ scores correlated

positively with overall mean scores from colleagues (R=.48)”Sargeant et al.

(2003)


The ironic nature of self-assessment

• We can’t trust it … but we should encourage it

• We need to influence it … without changing it

Self Efficacy• Influenced by feedback, but also influences

future performance

• Not merely a passive reflection of performance, but part of a self-fulfilling prophecy that affects performance

(see Dweck, 1999)

Implications for self-regulation

• “Rose colored glasses” approach to reflection understandable and necessary• Important for ability to function and succeed

• Can never create a “good self-assessor” • And probably shouldn’t try



2. Self-assessment is insufficient3. Self-assessment is vital4. The goal is to enable

discovery

Age Diagnosis/ health risk

Lab results Medication use

0102030405060708090

Type of patient information

% o

f phy

sici

ans

% of physicians for whom it is difficult/impossible to generate

practice data about…

(Audet, et al., 2005)

One 2 to 9 10 to 49 50 or more05

101520253035404550

Number of physicians per practice

% o

f phy

sici

ans

% of physicians who have access to any quality of care data

(Audet, et al., 2005)

The need for difficulties• Learners tend to request feedback after

successful experiences(Chiviacowsky and Wulf, 2002)

• Learning strategies perceived as beneficial are often sub-optimal

(Simon and Bjork, 2001)• Proficiency tends to be inferred from the

rate of learning rather than performance(Eva, 2009)

Jowett, et al. (2007)

Learner

Material

Test-enhanced learning

5 Minutes 2 Days 1 Week0

0.10.20.30.40.50.60.70.80.9

Study, StudyStudy, Test

Retention Interval

Perc

ent R

ecal

l

See Larsen, Butler, and Roediger (2008)

“The phrase self-assessment should not imply an isolated or individualistic activity; it should commonly involve peers, teachers, and other sources of information”

(Boud, 1999)

“Education is learning what you didn’t even know you didn’t know”

(Boorstin, 1914)




2. Self-assessment is insufficient3. Self-assessment is vital4. The goal is to enable discovery5. Numbers can be harmful

“Feedback interventions that draw attention to

the self via normative cues are largely ineffective”

(Kluger and DeNisi, 1998)


“Studies that augmented

feedback with consultation

produced substantially larger

differences, but other

methodological variations had little

effect”(Marsh and Roche,

1997)



1. There are no universal principles2. Self-assessment is insufficient3. Self-assessment is vital4. The goal is to enable discovery5. Numbers can be harmful6. Credibility is likely to be the key

Credibility

“With your clinical partner … they know your strengths and weaknesses and they’re probably going to give you quite useful feedback. Maybe a consultant watched you one time. They might not have the whole picture of what you’re capable of and what you normally do”

(Sargeant, et al., 2010)

Credibility

Of relationships: Engagement and awareness

Of the process: Validity and authenticity

Of intent: Beneficence and non-maleficence

(Sargeant, et al., 2010)

Thanks

[email protected]

Kruger and Dunning (1999)

Kruger and Dunning (1999)

• Self-assessment broadly conceived of as a prequel to self-directed learning activities aimed at performance improvements

• With respect to patient safety, we’re much more concerned about situation specific awareness (i.e., “reflection in practice”)• Continuous monitoring of whether or not one

has sufficient skills to effectively solve the current problem

The self-assessment literature(as summarized by Eva and Regehr, 2008)

Self-monitoringKnowing when to defer• Participants clearly

knew when they were likely to respond in error

Round 1 Round 20

1020304050607080

Perc

ent

Corr

ect

(Eva and Regehr, 2007)

Self-monitoring

(Eva and Regehr, 2007)

Self-monitoring•Self-assessment (as overall judgment of ability) is importantly different from self-awareness (monitoring)

•Difficulty with self-assessment is not that we can’t judge our performance, but that we don’t aggregate well over events

Documents

Principles of feedback delivery and the role of self- assessment