Systematic reviews of health promotion and

public health interventions

Rebecca Armstrong

Elizabeth Waters

Cochrane Health Promotion & Public Health Field

Overview

Overview of systematic reviews Outline of The Cochrane Collaboration Role of the HPPH Field Function of systematic reviews in informing

policy and practice Key elements of systematic reviews

Asking answerable questions Searching for evidence Assessing quality Synthesising results Applicability and transferability

Meta-analysis

Systematic reviews

Reviews(narrative/literature/traditional)

Types of reviews

Narrative reviews

Usually written by experts in the fieldUse informal and subjective methods to

collect and interpret informationUsually narrative summaries of the

evidence

Read: Klassen et al. Guides for Reading and Interpreting Systematic Reviews. Arch Pediatr Adolesc Med 1998;152:700-704.

What is a systematic review?

A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyse data from the studies that are included in the review*

*Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

Key elements of a systematic review

Structured, systematic process involving several steps :

1. Formulate the question2. Plan the review3. Comprehensive search4. Unbiased selection and abstraction process5. Critical appraisal of data6. Synthesis of data (may include meta-analysis)7. Interpretation of results

All steps described explicitly in the review

Systematic vs. Narrative reviews

Scientific approach to a review article

Criteria determined at outset

Comprehensive search for relevant articles

Explicit methods of appraisal and synthesis

Meta-analysis may be used to combine data

Depend on authors’ inclination (bias)

Author gets to pick any criteria

Search any databases

Methods not usually specified

Vote count or narrative summary

Can’t replicate review

Advantages of systematic reviews

Reduce biasReplicabilityResolve controversy between conflicting

studiesIdentify gaps in current researchProvide reliable basis for decision making

Limitations of systematic reviews specific to health promotion

Results may still be inconclusiveThere may be no trials/evidenceThe trials may be of poor qualityThe intervention may be too complex to be

tested by a trialPractice does not change just because

you have the evidence of effect/effectiveness

Consider these interventions…

Interventions to promote smoke alarm ownership and functionSchool-based driver education for the prevention of traffic crashesHelmets for preventing head and facial injuries in bicyclists

Do you think the results identified in SRs will be good, promising or absent (and potentially harmful)?

Results from systematic reviews

Helmets reduce bicycle-related head and facial injuries for bicyclists of all ages involved in all types of crashes including those involving motor vehicles.

The results provide no evidence that drive education reduces road crash involvement, and suggest that it may lead to a modest but potentially important increase in the proportion of teenagers involved in traffic crashes.

Results from this review suggest that area-wide traffic calming in towns and cities may be a promising intervention for reducing the number of road traffic injuries and deaths. However, further rigorous evaluations of this intervention are needed.

The Cochrane Collaboration

International non-profit organisation that prepares, maintains, and disseminates systematic up-to-date reviews of health care interventions

Cochrane Collaboration

Named in honour of Archie Cochrane, a British researcher

In 1979:

“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials”

The Cochrane Library

Cochrane Systematic reviews : Cochrane reviews and protocols

Database of Reviews of Effects: Other systematic reviews appraised by the Centre for Reviews and Dissemination.

Cochrane Central Register of Controlled Trials:

Bibliography of controlled trials (some not indexed in MEDLINE).

Health Technology Assessment Database: HTA reports

NHS Economic evaluation database:

Economic evaluations of health care interventions.

The Cochrane Library

www.thecochranelibrary.com

Cochrane HPPH Field

Represent the needs and interests of those in health promotion and public health in Cochrane matters

Represent Cochrane in health promotion and public health forums

In transition from Field to Review Group Will edit PH reviews for the Cochrane Library

Cochrane Collaboration Structure

Steering Group

Review Groups

Centres Fields

Methods GroupsConsumer Network

Collaborative Review Groups (50)

Produce systematic reviews relevant to a particular disease or health issue for inclusion in the Cochrane Library

Examples Airways Group Drug and Alcohol Group Heart Group Injuries Group Skin Group Pregnancy and Childbirth Group Stroke Group Breast Cancer Group

Methods Groups (12)

Provide advice and support in the development of the methods of systematic reviews

Examples Non-Randomised Studies Screening and Diagnostic Tests Empirical Methodological Studies Qualitative Methods

Cochrane Centres (14)

Work to assist all Cochrane entities within a specific geographical area

Examples Australasian Cochrane Centre (at Monash) South African Cochrane Centre Italian Cochrane Centre Chinese Cochrane Centre

Cochrane Fields/Networks (9)

Represent an area of interest which spans a number of health problems - and hence a number of Review Groups

Examples Health Promotion and Public Health Field Primary Health Care Field Cancer Network Child Health Field

Cochrane HPPH Field

Cochrane Fields represent a population group, or type of care that overlaps multiple Review Group areas

HPPH Field Registered in 1996 Administered from Melbourne Funded by VicHealth Over 400 members on contact database across

>30 countries

Staff

Elizabeth Waters (Director) Jodie Doyle (Coordinator) Rebecca Armstrong (Senior Research Fellow) Naomi Priest (Research Fellow)

Asking an answerable Asking an answerable questionquestion

Questions of interest

Effectiveness: Does the intervention work/not work? Who does it work/not work for?

Other important questions: How does the intervention work? Is the intervention appropriate? Is the intervention feasible? Is the intervention and comparison relevant?

Answerable questionsAnswerable questions

EFFECTIVENESS

A description of the populations P

An identified intervention I

An explicit comparison C

Relevant outcomes O

A PICO questionA PICO question

Time-consuming question:

What is the best strategy to prevent smoking in young people?

An answerable questionAn answerable question

Q. Are mass media (or school-based or community-based) interventions effective in preventing smoking in young people?

Problem, population

Intervention Comparison Outcome Types of studies

Young people under 25 years of age

a) Televisionb) Radioc) Newspapers d) Bill boardse) Postersf) Leafletsg) Booklets

a) School-based interventions

b) No intervention

a) objective measures of smoking (saliva thiocyanate levels, alveolar CO)

b) self-reported smoking behaviour

c) Intermediate measures (intentions, attitude, knowledge, skills)

d) Media reach

a) RCT

b) Controlled before and after studies

c) Time series designs

The PICO(T) chartThe PICO(T) chart

Finding the evidenceFinding the evidence

Systematic review processSystematic review process

1. Well formulated question

2. Comprehensive data search

3. Unbiased selection and abstraction process

4. Critical appraisal of data

5. Synthesis of data

6. Interpretation of results

A good searchA good search

Clear research question Comprehensive search

All domains, no language restriction, unpublished and published literature, up-to-date

Document the search (replicability)

Components of electronic searching

1. Describe each PICO component2. Start with primary concept3. Find synonyms

a) Identify MeSH / descriptors / subject headingsb) Add textwords

4. Add other components of PICO question to narrow citations (may use study filter)

5. Examine abstracts6. Use search strategy in other databases

(may need adapting)

So you want to do a ‘quick & dirty’?

DARE CENTRAL PubMed (clinical queries, related records) CDC NICE Organisations who do work in your area …google

The Cochrane Library: The Cochrane Library: www.thecochranelibrary.comwww.thecochranelibrary.com

Cochrane HPPH Field

Health-evidence.ca

The Guide to Community Preventive Services

http://www.thecommunityguide.org/

National Institute for Health and Clinical Excellence

http://www.publichealth.nice.org.uk/page.aspx?o=home

Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre)

http://eppi.ioe.ac.uk

Effective Public Health Practice Project (EPHPP)

http://www.myhamilton.ca/myhamilton/CityandGovernment/HealthandSocialServices/Research/EPHPP/

Centre for Reviews and Dissemination

http://www.york.ac.uk/inst/crd

Other sources of primary researchOther sources of primary research

Searching

www.google.com.au The order of terms will effect the results so

start with the obvious or key concept No need for ‘and’ Google will ignore common words

If they are important use + (e.g. policy + 3) Phrase searching is useful eg “suicide

prevention” Google searches for variations on words eg

diet, dietary

Searching

Where terms have multiple meanings you can direct google to remove sites you want to avoid (e.g. bass –music)

Keep your search strings brief Mental health promotion initiatives to prevent suicide

in young people

Compartmentalise your search strings “mental health promotion” suicide “Suicide prevention” “young people” Prevention and suicide and youth

Searching

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi Use the same principles for google – keep it

short and sweet. Key features

Journals DatabaseMeSH DatabaseSingle Citation MatcherClinical Queries

Searching

Searching

Searching

Select the Limits tab – just under the search string

Searching

These next few slides show you how to search MeSH terms in PubMed. Useful if you don’t have access to electronic databases. You combine this method with the one for text words outlined above. The process for combining text words and MeSH terms is outlines below.

ExampleExample

Mass media interventions to prevent smoking in young people

P= Young people

STEP ONE:STEP ONE:Find MeSH and textwords to

describe young people

ExampleExample

Mass media interventions to prevent smoking in young people

P= Young people

MeSH: Adolescent

Child

Minors

ExampleExample

Mass media interventions to prevent smoking in young people

P= Young people

Textwords: Adolescent GirlChild BoyJuvenile TeenagerYoung people Young adultStudent Youth

Textwords

Truncation $:

To pick up various forms of a word

Teen$.tw Smok$.twTeenage Smoke

Teenager Smoking

Teenagers Smokes

Teens Smoker

Teen Smokers

Textwords

Wild cards ? and #:

To pick up different spellings

Colo?r.tw (? Can be substituted for one or no characters)

Colour

Color

Wom#n.tw (# Substitutes for one character)

Woman

Women

Textwords

Adjacent ADJn: retrieves two or more query terms within n words of each other, and in any order Great when you are not sure of phraseologyEg sport adj1 policy

Sport policyPolicy for sport

Eg mental adj2 healthMental healthMental and physical health

Example continuedExample continued

Mass media interventions to prevent smoking in young people

I = Mass media interventions

STEP TWO:STEP TWO:Find MeSH and textwords to

describe mass media interventions

Example continuedExample continued

MeSH Mass media Audiovisual aids Television Motion pictures Radio Telecommunications Newspapers Videotape recording Advertising

Example continuedExample continued

Mass media interventions to prevent smoking in young people

O = Prevention of smoking

STEP THREE:STEP THREE:Find MeSH and textwords to

describe prevention of smoking

Example of searchExample of search

P = YOUNG PEOPLEP = YOUNG PEOPLE

MeSHMeSH TextwordsTextwords

……………………………………………….. ……………………….……………………….……………………….………………………. ……………………….……………………….

……………………………………………….. ……………………….……………………….

I = MASS MEDIAI = MASS MEDIA

MeSHMeSH TextwordsTextwords

……………………………………………….. ……………………….……………………….……………………….………………………. ……………………….……………………….

……………………………………………….. ……………………….……………………….

C = (if required)C = (if required)

O = PREVENTION OF SMOKINGO = PREVENTION OF SMOKING

MeSHMeSH TextwordsTextwords……………………………………………….. ……………………….……………………….

……………………….………………………. ……………………….……………………….……………………………………………….. ……………………….……………………….

OROR

OROR

OROR

PP

ANDAND

II

ANDAND

CC

ANDAND

OO

Different bibliographic databasesDifferent bibliographic databases

Databases use different types of controlled vocabulary Same citations indexed differently on

different databases Medline and EMBASE use a different

indexing system for study type PsycINFO and ERIC do not have specific

terms to identify study typesNeed to develop search strategy for each database

Study design filters

RCTs See Cochrane Reviewer’s Handbook

Non-RCTs Not yet developed, research in progress

Qualitative research Specific subject headings used in CINAHL, ‘qualitative research’

used in Medline CINAHL Filter: Edward Miner Library

http://www.urmc.rochester.edu/hslt/miner/digital_library/tip_sheets/Cinahl_eb_filters.pdf

Systematic reviews/meta-analyses CINAHL: as above Medline

http://www.urmc.rochester.edu/hslt/miner/digital_library/tip_sheets/OVID_eb_filters.pdf

Medline and Embasehttp://www.sign.ac.uk/methodology/filters.html

PubMed

2. Unpublished literature2. Unpublished literature

Not all known published trials are identifiable in Medline (depending on topic)

Only 25% of all medical journals in Medline Non-English language articles are under-

represented in Medline (and developing countries)

Publication bias – tendency for investigators to submit manuscripts and of editors to accept them, based on strength and direction of results (Olsen 2001)

2. Unpublished literature2. Unpublished literature

Hand searching of key journals and conference proceedings

Scanning bibliographies/reference lists of primary studies and reviews

Contacting individuals/agencies/ academic institutions

Neglecting certain sources may result in reviews being biased

Librarians are your friends!Librarians are your friends!

Principles of critical Principles of critical appraisalappraisal

Critical appraisalCritical appraisal

The process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.

Alison Hill, Critical Appraisal Skills Programme, Institute of Health

Sciences, Oxford http://www.evidence-based-medicine.co.uk

Critical appraisal I: Critical appraisal I: Quantitative studiesQuantitative studies

Why appraise validity?Why appraise validity?

Not all published and unpublished literature is of satisfactory methodological rigour Just because it is in a journal does not mean it

is sound! Onus is on you to assess validity!

Quality may be used as an explanation for differences in study results

Guide the interpretation of findings and aid in determining the strength of inferences

Bias – quality assessment toolBias – quality assessment tool

1. Selection bias

2. Allocation bias

3. Confounding

4. Blinding (detection bias)

5. Data collection methods

6. Withdrawals and drop-outs

7. Statistical analysis

8. Intervention integrity

Recruit participants

Allocation

Exposed to intervention

Not exposed to intervention

Follow-upFollow-up

Outcome Outcome

Allocation of concealment

Blinding of outcome assessment

Intention-to-treat

Withdrawals

Data collection methods

Selection bias

Integrity of intervention

Intervention Control

Analysis AnalysisStatistical analysis

Confounding

Critical appraisal tools

RCTs The Quality Assessment Tool for Quantitative

Studies (http://www.city.hamilton.on.ca/PHCS/EPHPP/).

Non-RCTs Cochrane Effective Practice and Organisation

of Care Group (http://www.epoc.uottawa.ca/). The Berkeley Systematic Reviews Group

(http://www.medepi.net/meta/)

Critical appraisal II: Critical appraisal II: Qualitative studiesQualitative studies

Qualitative researchQualitative research

… explores the subjective world. It attempts to understand why people behave the way they do and what meaning experiences have for people.

Qualitative studies of experienceProcess evaluation

Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

CASP appraisal checklistCASP appraisal checklist

1. Clear aims of research (goals, why it is important, relevance)

2. Appropriate methodology (what, how, why)3. Sampling strategy4. Data collection5. Relationship between researcher and

participants6. Ethical issues7. Data analysis8. Findings9. Value of research (context dependent)

Other qualitative checklistOther qualitative checklist

Quality framework Government Chief Social Researcher’s Office,

UK http://www.strategy.gov.uk/files/pdf/Quality_framew

ork.pdf

19 question checklist for process evaluations (EPPI-Centre)

Appraisal of a systematic reviewAppraisal of a systematic review

10 questions1. Clearly-focused question2. The right type of study included3. Identifying all relevant studies4. Assessment of quality of studies5. Reasonable to combine studies6. What were the results7. Preciseness of results8. Application of results to local population9. Consideration of all outcomes10. Policy or practice change as a result of evidence

CASP

Interpretation of resultsInterpretation of results

Factors influencing effectivenessFactors influencing effectiveness

Type III error (integrity of intervention)Theoretical framework of interventionContext

For whom did the intervention work, why, in what circumstances, at what cost

Difficulties addressing inequalitiesDifficulties addressing inequalities

Studies rarely present information on differential effects of interventions

Cannot locate studies addressing inequalities

May need original data from authorsLow power to detect subgroup differences

Integration of evidence into Integration of evidence into practicepractice

Assessing the applicability and transferability of interventions

Applicability – whether the intervention process could be implemented in the local setting, no matter what the outcome is. Is it possible to run this intervention in this local

setting? Eg. provision of condoms in area where they

are not acceptable for religious reasons

Wang et al 2005

Assessing the applicability and transferability of interventions

Transferability – if the intervention were to be implemented in the local setting, would the effectiveness of the program be similar to the level detected in the study setting? E.g. if the interventionists lack experience and

have few skills in delivering the intervention then its effectiveness in the local setting may be lower than that demonstrated in the study setting

Wang et al 2005

Review of questions proposed by Wang et al

Contact details

Rebecca Armstrong rarmstrong@vichealth.vic.gov.au 03 9667 1336 If I can’t help you I might be able to point you in

the right direction. If you are interested in training and support for

conducting SRs or increasing uptake within your organisation, region, state please let me know