Approaches to evaluating Approaches to evaluating Clinical Librarian Services: Clinical Librarian Services:

a systematic reviewa systematic reviewa systematic reviewa systematic reviewPresented by: Presented by: ØØ Alison Brettle, University of Salford, Salford, UK Alison Brettle, University of Salford, Salford, UK ØØ Anne Webb, The Christie NHS Foundation Trust, Manchester, UKAnne Webb, The Christie NHS Foundation Trust, Manchester, UK

On behalf of NW England Clinical Librarians Systematic Review GroupOn behalf of NW England Clinical Librarians Systematic Review Group

Partly funded by North West Health Care Libraries UnitPartly funded by North West Health Care Libraries UnitLibrary and Information Health Network North WestLibrary and Information Health Network North West

NHS and the North WestNHS and the North West

10 Strategic Health Authorities (SHA)

Department of Health

NW SHA

Nationwide employs: • around 90,000 hospital doctors• 35,000 general practitioners (GPs)• 400,000 nurses • 16,000 ambulance staff (www.nhs.uk)

10 Strategic Health Authorities (SHA)NW SHA

Acute Trusts, Specialist Trusts, Mental Health Trusts, Primary Care Trusts (PCT) Ambulance TrustsAcute Trusts, Specialist Trusts, Mental Health Trusts, Primary Care Trusts (PCT) Ambulance Trusts

The North West covers the 5 areas of Greater Manchester, Merseyside, Cheshire, Cumbria and Lancashire with a population of 6.8 million.

Approximately 8% of people work in the healthcare sector – including a team of librarians interested in clinical librarianship!

UK North West Clinical Librarian UK North West Clinical Librarian Systematic Review Group:Systematic Review Group:

ØØ Lucy Anderson, NHS BuryLucy Anderson, NHS BuryØØ Alison Brettle, University of SalfordAlison Brettle, University of SalfordØØ Michelle MadenMichelle Maden--Jenkins, Edge Hill UniversityJenkins, Edge Hill UniversityØØ Rosalind McNally, National Primary Care Research and Rosalind McNally, National Primary Care Research and

Development Centre, University of ManchesterDevelopment Centre, University of Manchester

ØØ Tracey Tracey PratchettPratchett, University Hospitals of , University Hospitals of Morecambe Bay NHS Trust Morecambe Bay NHS Trust

ØØ Jenny Jenny TancockTancock, University Hospitals of , University Hospitals of Morecambe Bay NHS TrustMorecambe Bay NHS Trust

ØØ Debra Thornton, Blackpool, Fylde and Wyre Debra Thornton, Blackpool, Fylde and Wyre NHS Foundation Trust NHS Foundation Trust

ØØ Anne Webb, The Christie NHS Foundation Anne Webb, The Christie NHS Foundation TrustTrust

DriversDrivers

ØØ The Hill Report (2008) The Hill Report (2008) –– includes includes recommendations that librarians evaluate their recommendations that librarians evaluate their servicesservices

ØØ Measuring impact of services is a challengeMeasuring impact of services is a challengeØØ Measuring impact of services is a challengeMeasuring impact of services is a challenge

ØØ Previous research demonstrated variability in Previous research demonstrated variability in models and limited effectiveness of Clinical models and limited effectiveness of Clinical Librarian ServicesLibrarian Services

What’s Known to Date?What’s Known to Date?ØØ 3 other reviews3 other reviews

ll Winning & Beverley (2003Winning & Beverley (2003))ll CimplCimpl Wagner & Byrd (2004)Wagner & Byrd (2004)ll WeightmanWeightman & Williamson (2005)& Williamson (2005)

ØØ Identified Weaknesses Identified Weaknesses

ll Small sample sizeSmall sample sizell Low response rateLow response ratell Response bias Response bias ll Poor reporting/inadequate information on methodsPoor reporting/inadequate information on methodsll Researcher bias/desirability bias Researcher bias/desirability bias ll NonNon--specific patient care outcomesspecific patient care outcomesll NonNon--use of reliable/valid methodsuse of reliable/valid methodsll Subjective reporting of resultsSubjective reporting of resultsll Results less likely to be quantifiedResults less likely to be quantified

Overall AimOverall Aim

ØØ To update previous reviewsTo update previous reviews

ØØ Provide guidance for future evaluations.Provide guidance for future evaluations.

ØØClarification of the models of clinical Clarification of the models of clinical librarianship and how best to measure librarianship and how best to measure impact of services in relation to these?impact of services in relation to these?(Hill Report) (Hill Report)

Objectives of Our StudyObjectives of Our Study

1.1. To determine which models of Clinical Librarian To determine which models of Clinical Librarian services have been evaluatedservices have been evaluated

2.2. To determine whose perspective has been To determine whose perspective has been 2.2. To determine whose perspective has been To determine whose perspective has been evaluatedevaluated

3.3. To determine what outcome To determine what outcome measuresmeasures have have been usedbeen used

4.4. To determine the quality of the methods usedTo determine the quality of the methods used

Methods 1Methods 1-- SearchingSearchingØØ Searched 20 databases from 2001 onwardsSearched 20 databases from 2001 onwards

ØØ Scanned references and hand searched 2 journals Scanned references and hand searched 2 journals HILJ, JMLAHILJ, JMLA

ØØ Google searchGoogle search

ØØ Grey literature Grey literature –– mail lists, known contacts, CILIP mail lists, known contacts, CILIP Update etc. Update etc.

ØØ BibliographiesBibliographies

Methods 2 Methods 2 –– Filtering and Filtering and ExtractionExtraction

ØØ Filtering and article selectionFiltering and article selectionØØ Initial filter of irrelevant articles undertaken by 2 Initial filter of irrelevant articles undertaken by 2 people.people.

ØØ Scanned titles and abstracts (in pairs) Scanned titles and abstracts (in pairs) ØØ Scanned titles and abstracts (in pairs) Scanned titles and abstracts (in pairs) ØØ Obtained full papers (checked in pairs)Obtained full papers (checked in pairs)ØØ Developed and evaluated critical appraisal/data Developed and evaluated critical appraisal/data extraction toolextraction tool

ØØ Extracted relevant papers (in pairs)Extracted relevant papers (in pairs)ØØ Data management tools: Data management tools: RefworksRefworks and Excel and Excel

Inclusion CriteriaInclusion CriteriaØØ Studies that meet the Hill definitionStudies that meet the Hill definition

ØØ Studies which are described as outreach but the focus is Studies which are described as outreach but the focus is to support patient careto support patient care

ØØ Studies which describe services providing patient Studies which describe services providing patient ØØ Studies which describe services providing patient Studies which describe services providing patient information information –– evaluation outcome relating to patient careevaluation outcome relating to patient care

ØØ Published post 2001Published post 2001

ØØ Reports evaluation methodologyReports evaluation methodology

ØØ English languageEnglish language

Results 1Potentially relevant citations identified

n=2040

Excludedn = 857

Initial assessment of titles and abstractsn = 456

Excludedn = 62

Full text assessment –potentially relevant items

n = 91

Papers going forward to appraisal/ data extraction n= 29

Final Inclusionn = 21

Breakdown by Country

Country No.

UK 15

US 5

Holland 1

Breakdown by Study DesignStudy design No. of papers

Survey 16

Qualitative 10

Experimental (e.g. RCT) 1

Quasi-experimental (e.g. Pre-post test) 1

Service evaluation 13Service evaluation 13

Action Research 2

Case Study 7

Other 5

Unclear 0

Not stated 0

Breakdown of Service Models

Information at the point of need

Information at the point of need plus critical appraisal and synthesis

Question and Answer Question and Answer Question and Answer service1

Question and Answer service plus critical appraisal and synthesis3

Outreach14

Outreach plus critical appraisal and synthesis= informationist4

Breakdown of OutcomesOutcome criteria

Usage statistics 62%

Use of information 52%

Relevance of results 48%

Usefulness of results 48%48%

Time saving 48%

General impact on patient care 38%

Time to respond 33%

Other 24%

Improvement in information literacy 23%

Improvement in confidence 4%

Cost 0%

Not stated 0%

Breakdown of Perspective

PerspectiveNo. of papers

User 16

Librarian 7

Library Service 11

Organisation 3

Unclear 1

Breakdown of QualityQuality measure (Weightman et al) no. %

Appoint researchers who are independent of the library service 5 23

Ensure that all respondents are anonymous and that they areaware of this.

4 19

Survey all members of chosen user group(s) or a randomsample

12 57sample

Agree a set of questions that are objective, well used in previousresearch, and developed with input from library users.

9 42

Use the critical incident technique. 6 29

Combine a questionnaire survey with a smaller, but also randomsample of follow-up interviews.

8 38

ConclusionConclusion

1.1. Models of Clinical Librarian Service Models of Clinical Librarian Service –– 4 4 models evaluatedmodels evaluated

2.2. Evaluation has been mainly from a Evaluation has been mainly from a service and service user perspectiveservice and service user perspectiveservice and service user perspectiveservice and service user perspective

3.3. A wide range of outcome measures have A wide range of outcome measures have been usedbeen used

4.4. Quality has improved, but needs to Quality has improved, but needs to improve furtherimprove further

Thank You!Thank You!

ll Thank you for listeningThank you for listeningll Thank you to the NW Clinical Librarian Systematic Thank you to the NW Clinical Librarian Systematic Review GroupReview Group

ll Thank you to North West Health Care Libraries Unit Thank you to North West Health Care Libraries Unit for fundingfor fundingfor fundingfor funding

Any questions?Any questions?Contacts: Contacts: aa..brettle@salford.ac.ukbrettle@salford.ac.uk

aannne.webb@christie.nhs.ukne.webb@christie.nhs.uk

ReferencesReferencesØØ Cimpl Wagner, K. and Byrd, G.D. Evaluating the effectiveness of clinical medical Cimpl Wagner, K. and Byrd, G.D. Evaluating the effectiveness of clinical medical

librarian programs: a systematic review of the literature. librarian programs: a systematic review of the literature. Journal of the Medical Journal of the Medical Library AssociationLibrary Association 92 (1):1492 (1):14--33, 2004.33, 2004.

ØØ Hill, P. Report of a national review of NHS library services in England: from Hill, P. Report of a national review of NHS library services in England: from knowledge to health in the 21st Century, National Library for Health, 2008. Available knowledge to health in the 21st Century, National Library for Health, 2008. Available from: http://www.library.nhs.uk/aboutnlh/review (accessed 5th May 2008)from: http://www.library.nhs.uk/aboutnlh/review (accessed 5th May 2008)

ØØ Rankin, J. A. et al .The emerging Informationist Specialty: a systematic review of the Rankin, J. A. et al .The emerging Informationist Specialty: a systematic review of the ØØ Rankin, J. A. et al .The emerging Informationist Specialty: a systematic review of the Rankin, J. A. et al .The emerging Informationist Specialty: a systematic review of the literature JMLA 96 (3): 194 literature JMLA 96 (3): 194 -- 206, 2008206, 2008

ØØ Weightman, AL and Williamson, J. The value and impact of information provided Weightman, AL and Williamson, J. The value and impact of information provided through library services for patient care: a systematic review, through library services for patient care: a systematic review, Health Information and Health Information and Libraries JournalLibraries Journal, 22: 4, 22: 4--25, 200525, 2005

ØØ Weightman AL, Urqhuart C, Spink S, Thomas R. The value and impact of information Weightman AL, Urqhuart C, Spink S, Thomas R. The value and impact of information provided through library services for patient care: developing guidance for best provided through library services for patient care: developing guidance for best practice. practice. Health Information and Libraries Journal Health Information and Libraries Journal 2008;26:632008;26:63--71.71.

ØØ Winning, M.A. and Beverley, C.A. Clinical librarianship: a systematic review of the Winning, M.A. and Beverley, C.A. Clinical librarianship: a systematic review of the literature. literature. Health Information & Libraries JournalHealth Information & Libraries Journal 20: Suppl20: Suppl--21, 2003.21, 2003.

Glossary

Ø CILIP - Charted Institute of Library and Information Professionals

Ø DH - Department of HealthØ HCLU - North West Health Care Libraries UnitHILJ - Health information and Libraries JournalØ HILJ - Health information and Libraries Journal

Ø JMLA - Journal of the Medical Library AssociationØ LIHNN - Library and Information Health Network North West (England)

Ø NHS - National Health ServiceØ PCT - Primary Care TrustØ SHA - Strategic Health Authority