What is the evidence in evidence-based practice: citation analysis of references in Australian cancer clinical guidelines

Berenika M. Webster, Thomson ReutersDanielle Penn, University of Sydney

International Congress on Medical Librarianship Brisbane 9 September 2009

2

Agenda

• Measuring impact of research

• Clinical guidelines

• Our study

What do we want to evaluate?

Researchperformance

Academic impact

Economic impact

Societal impact

Research process: outputs, impacts and outcomes

Applications

for funds

New

vaccines

Better

diagnosis

New drugs

Less illness incidence

Better clinical care

Papers at

conference

s

Popular

opinions

Pollution

regulations

Lifestyle

choices

Physical

environment

Citations in

papers

Papers in journals

Patents

Medical

education

Public health

campaigns Government

policy

Clinical

guidelines

Decisions

on funding

Conduct of

research

Training of

researchers

Newspaper

articles

Clinical

trials

Economic

output Other docs,

books

Research problem

After G. Lewison, 1999

5

Clinical guidelines• Definition

“systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances” (Field and Lohr, 1990)

• Aimto improve the quality of healthcare and outcomes for

patients […] at both clinical level and promotion of efficient allocation of resources and better delivery systems (Shekelle, 1999)

• Controversial

6

Previous studies• Grant et al – BMJ, 2000

• Lewison and Wilcox-Jay – Proc 9th Intl Conf on Scientometrics and Informetrics, Beijing 2003

• Webster, Lewison and Rowlands- Mapping the Landscape II, 2003

• Sullivan and Lewison – Br Jnl of Cancer, 2008

7

Previous studies - findings• Studied British guidelines (NICE for England and

Wales and SIGN for Scotland)• Age distribution is comparable with that of the

population• British research is overcited (by 300%); research

from Japan and developing countries is mostly ignored

• Applied research is cited more extensively• Cited references come from high impact journals• Ratio of funded papers is twice that of population

8

Clinical guidelines in Australia• 47 current practice guidelines from NHMRC• 85 published by Medical Journal of Australia

– Clinical guidelines published by the MJA represent the consensus opinion of experts based on review of the scientific literature

• Subject areas covered: – Aboriginal Health , Cardiology,  Endocrinology , General

medicine, Geriatrics, Haematology, Immunology and Allergy, Infectious Diseases, Nutrition, Obstetrics and Gynaecology and Women's Health , Paediatrics, Palliative Care, Psychiatry, Respiratory Medicine, Renal Medicine, Rheumatology

9

Research question:• What are the characteristics of journal literature

cited on Australian cancer clinical guidelines?

– What is the use of “local” literature?

– Is applied research cited more often on clinical guidelines?

– What is the impact of literature cited on clinical guidelines?

10

Method• 19 Australian cancer clinical guidelines• 4,700 cited references to journal literature identified• Bibliometric analyses of these references

conducted– Descriptive bibliometrics (volume, age and subject areas)– Advanced bibliometrics (normalised indicators of impact)

• Against subject category• Against journal set (C-index)

• Web of Science used as source of citation and benchmarking data

11

Time distribution of references on Australian cancer guidelines, WoS 1981-2008 data

12

Document types referenced on Australian cancer guidelines

Document type Papers Citations CPP Median

Article 3,746 363,419 97.02 45

Proceedings Paper 388 27,937 72.00 38

Review 380 44,820 117.95 42

Editorial 94 3,961 42.14 16

Letter 53 2,800 52.83 7

Note 50 3,349 66.98 39

Other 17 117 6.9

Total 4,728 446,403 94.42 42

13

Subject distribution of references on Australian cancer clinical guidelines (percentages), WoS data 1981-2008

ON

CO

LOG

Y

SU

RG

ER

Y

ME

DIC

INE

, GE

NE

RA

L &

INTE

RN

AL

RA

DIO

LOG

Y, N

UC

LEA

R M

ED

ICIN

E &

ME

DIC

AL

IMA

GIN

G

CLI

NIC

AL

NE

UR

OLO

GY

DE

RM

ATO

LOG

Y

PU

BLI

C, E

NV

IRO

NM

EN

TAL

& O

CC

UP

ATI

ON

AL

HE

ALT

H

HE

MA

TOLO

GY

GA

STR

OE

NTE

RO

LOG

Y &

HE

PA

TOLO

GY

PA

THO

LOG

Y

05

10152025303540

14

Ratio of countries’ presence on references on Australian cancer clinical guidelines, WoS data1981-2008

Red line represents baseline

15

Australian papers on guidelines, distribution by subject category, WoS data 1981-2008

Primary subject Rank % Australian paps % Wld paps RatioDERMATOLOGY 2 9.7 1.3 7.8SURGERY 3 8.9 2.3 3.8ONCOLOGY 4 6.6 1.9 3.5MEDICINE, GENERAL & INTERNAL 3 10.9 3.2 3.4PATHOLOGY 2 8.4 2.6 3.2PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH 3 11.0 3.4 3.2GASTROENTEROLOGY & HEPATOLOGY 3 7.5 2.4 3.2RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING 8 2.8 1.3 2.2CLINICAL NEUROLOGY 9 3.5 2.4 1.5HEMATOLOGY 13 2.2 2.2 1.0

16

Performance of Australian papers referenced on Australian cancer clinical guidelines, WoS data 1980-2008

Country Number of papers Percentage Citations per paper Median h-index Avg percentileUSA 1,998 37.86 133.1 55 241 14.73UK 610 10.27 143.27 52 138 16.2

Australia 404 7.65 79.17 35 93 20.5Canada 267 5.06 165.5 59 98 15.2France 261 4.95 162.58 54 93 16.3

Germany 252 4.47 126.99 53 86 15.5Italy 227 4.3 142.18 42 78 19.9

Netherlands 218 4.13 162.05 64 86 14.8Sweden 144 2.73 124.8 55 64 14.42

17

Commercial addresses on papers cited on Australian cancer clinical guidelines

Percentage of corporate addresses

Number of journals referenced on

Australian clinical guidelines

PercentagePercentage of

all WoS journals

Ratio

<1 % 224 40.4 42.41 0.951-5% 279 50.4 41.67 1.215-20% 47 8.5 13.10 0.65>20% 4 0.7 2.82 0.26

Classification from R. Tijssen

18

Relative performance of references on Australian cancer clinical guidelines, by subject categories, WOS data 1981-2008

Red line represents baseline, or “world average” for subject area

19

Relative performance of references on Australian cancer clinical guidelines, by journal (C-index), WOS data 1981-2008

Red line represents baseline, or “world average” for journal set

20

Distribution of papers cited on Australian cancer clinical guidelines by IF quartile,WoS and JCR data

21

Conclusions• Papers cited on Australian cancer guidelines are

older than papers cited in oncology literature overall

• Australian-authored papers disproportionately play significant role

• Papers referenced on Australian cancer clinical guidelines are published in high impact journals

• Papers referenced on Australian cancer guidelines outperform cancer research papers overall (as benchmarked against “world average” for subject category and journal set)

22

Further research• The role of Australian papers

– parochialism or an important component of clinical medicine?

• Corporate address method in assessing research level of the paper – comparative analysis of three approaches: CHI

classification – Lewison and Paraje key word analysis and Tijssen address classification