A tool to assess clinical handover quality

A/Prof Malcolm Moore

ANU Rural Clinical School; USyd Medical School

A/Prof Chris Roberts

USyd Medical School

On behalf of the Broken Hill UDRH ECPP research group (USyd: Dr Sue Kirby; UAdel: Prof Jonathan Newbury; UoW: A/Prof David Garne)

Background

• Broken Hill UDRH and the Extended Clinical Placement Program

• Far West LHD clinics

• Remote nurses

• Royal Flying Doctor Service

• Medical students

• ‘…but surely you’re not putting students in that position!’

• Are there safety issues for patients or students?

RFDS SE: 640,000 km2

approx. 25,000 people

Wilcannia

Remote telephone clinical handover

• RN and/or supervised medical student assesses the patient

• Consults with RFDS doctor

• Carries out management plan – may include retrieval

This is a handover call:

The exchange between health professionals of information about a patient accompanying either a transfer of control over, or of responsibility for, the patient.

Cohen and Hilligoss, 2010

Components of handover

Seven Handover ‘framings’2

• The handover conversation• Information processing (the most commonly studied)

• A stereotypical narrative (and deviations from this)

• Resilience of the handover (in revealing errors in assumptions)

• The broader context• Accountability for actions

• Social interaction

• Networking with relevant people

• Cultural norms in the organisation2Patterson and Wears, 2010

ISBAR

• Identification

• Situation

• Background

• Assessment

• Recommendation

• Suitable for in- and out-of-hospital (Cf. other specific frameworks)

• Promoted and used in FWLHD

Handover assessment tools

• Method• Examine written handover sheets

• Direct observation

• Analysis of audio and video recordings

• Tools• Use checklists of pieces of information (most common)

• Measure items related to clinical reasoning

• Analyse patterns of communication

• Interview clinicians

Why develop a new tool for this study?

• Usable by clinicians in routine work

• Developed to match local practice and expectations

• There is scanty literature on remote handover and the role of students in handover

The tool development and pilot

• Focus group of clinicians and students• How is handover done?• What do I want from a handover?

Led to…

• Item development – adding an item on clinical reasoning to the ISBAR elements

• Global assessment item• Based on focus group data

• Assessment rubric• Work- based assessment: how much questioning was required for each item?

The Broken Hill Handover Assessment Tool (BHHAT)• Introduces self and position (Introduction)

• Identifies main problem (Situation)

• Gives appropriate history (Background)

• Gives appropriate examination/observations (Assessment)

• Makes logical assessment (clinical reasoning item)

• Makes a clear recommendation (Recommendation)• 6 items added to give checklist sub-total (6 x 0-3)

• Global Rating: How confident am I that I received an accurate picture of the patient?

• Global score recorded separately (0,1,2,3)

PLEASE TICK APPROPRIATE JUDGMENT

Not performed competently

Able to perform under firm direction

Able to perform under modest direction

Able to perform under minimal direction

Identifies self and position

Identifies main problem

Gives appropriate historyGives appropriate examination/observationsMakes logical assessmentMakes a clear recommendation Global RatingHow confident am I that I received an accurate picture of the patient?

Ungraded observation of additional factors impacting quality

PLEASE TICK APPROPRIATE JUDGMENT

Not performed competently

Able to perform under firm direction

Able to perform under modest direction

Able to perform under minimal direction

Identifies self and position Requires direct prompting to elicit name and position

Introduces themselves as an afterthought during handover or after a subtle hint.

Incompletely introduces themselves prior to commencing handover

Introduces themselves and their role prior to commencing handover

Identifies main problem Unable to identify the main problems

Identifies and prioritises the main problems after extended prompting

Identifies and prioritises the main problems with a few further questions being needed

Identifies and prioritises the main problems

Gives appropriate history

Gives appropriate examination/observations

Makes logical assessment

Makes a clear recommendation

Global RatingHow confident am I that I received an accurate picture of the patient?

Not at all confident I am confident but required extended questioning on several aspects

I am confident but required some further questioning

I am confident and required little or no questioning

Ungraded observation of additional factors impacting quality

Please Comment

Results

• 10 nurses, 15 medical students, 8 RFDS doctors

• Calls scored ‘live’ by clinicians and from recordings by academic panel: all calls scored by at least 2 raters

• 55 calls generated 132 assessments

• Difference b/w items: ‘main problem’ mean 2.50 (0.75); ‘makes recommendation’ mean 2.08 (0.97)

• No significant difference between RN and student scores: group mean of global score 2.3/3

• Low-complexity cases scored higher than medium/high-complexity

Reliability; Generalisability

• Reliability: how reproducible are the assessments with repeated use?Several types including inter-observer (hawks and doves), intra-observer (good day/bad day), test-retest Each of these sources of error must be tested separately

• GeneralisabilityUses all of the data to quantify all sources of error without multiple experimentsWe want to know ‘true variance’: the differences between assessees that are stable across different observers and situationsNeed to quantify the sources of ‘error variance’ to calculate how many observers and cases we need to make the assessment reliable. For example: the observer; the case; and all of the interactions between assessee, observer + case. Uses ANOVA.

Generalisability calculations

• G-coefficient calculated for: • Checklist sub-total and for global score

• Assessing individual calls and assessing individual callers (over multiple calls)

• For individual calls (checklist more reliable)• 77% of variance from the call or call*assessor interaction

• D-study gave G-coefficient of 0.73 with 4 assessors, 0.80 with 6 assessors

• For individual callers (global score more reliable)• More speculative because most callers only assessed on 2-3 calls

Feedback from clinicians and researchers

• All doctors reported the tool was easy to use

User friendly; quite straightforward

[the] main thing is to pick it up and use it

• Acceptable in the course of routine clinical work

Although…the form is fine, it’s just another thing that we have to do

Tool validity

• Content validity• Developed with local clinicians

• Based on existing framework (ISBAR) and consistent with handover literature

• Refined after initial pilot of tool

• Other evidence from results and feedback• Differentiation of low- and med/high-complexity cases

• RN and student scores reflective of qualitative data on handover quality from clinicians and assessors

Discussion

• ‘Introduction’ item often hard to score in this context• Patient details part of the routine RFDS data sheet

• Callers often known or were having their call returned by RFDS

• Examination findings not always relevant• Some cases were very straightforward and routine

• Difference in receiving doctor’s communication styles• A problem for rating from recordings, not so much when used ‘live’

Responses to feedback

• Consider removing ‘Introduction’ (therefore ‘SBAR’)

• Underlines the importance of consultation and training in any site –particularly relating to the issue of ‘style’. Are clinicians content to allow a competent handover to proceed without initial interruption?

What this study adds

• The BHHAT is a handover assessment tool that is acceptable to and feasible for use by working clinicians: not just a research tool

• It can be used where clinical decision-making is required or an assessment of clinical reasoning skills is sought

• It can be used to assess the quality of handover in an organisation and assist in identifying training needs

Future directions

• The tool shows promise in the assessment of students and clinicians in individual episodes of handover.

• It could be considered for use in other rural and remote sites

• It could also be considered in the larger hospital setting as a professional assessment tool

• Larger studies are needed to provide further evidence of reliability and validity

References

• Cohen MD, Hilligoss PB. The published literature on handoffs in hospitals: deficiencies identified in an extensive review. Qual Saf Health Care. 2010:19:493-497

• Patterson E, Wears R. Patient handoffs: standardized and reliable measurement tools remain elusive. Jt Comm J Qual Patient Saf. 2010:36:52-61

• Crossley J, Davies H, Humphris G, Jolly B. Generalisability: a key to unlock professional assessment. Medical education. 2002;36(10):972-8