META-ANALYSIS AND SYSTEMATIC REVIEWS IN QUALITATIVE RESEARCH: MISSION IMPOSSIBLE?

Qualitative Inquiry Group Seminar University of Toronto, March 27, 2009

1

Ellen MacEachen (Institute for Work & Health, University of Toronto)Scott Reeves (Li Ka Shing Knowledge Institute, University of Toronto)

Overview – 4 parts

1. Where did systematic reviews come from & why do we do them?

2. Brief overview of the systematic review process

3. Cases that draw out systematic review issues Case I (Scott) Case II (Ellen)

4. Reflections on the conduct and usefulness of systematic reviews of qualitative studies

2

1. What are systematic reviews? Positivistic roots / standardization Synthesis of primary studies Multiple check points / dual reviewers /

transparency Assess quality of evidence (only include

‘rigorous’ studies) Inference through statistical analysis

Newer Qual types (meta-ethnography… )

3

Why do SRs?

Useful: Knowledge translation (policymakers/

practitioners) Initial entry into field (academics) Help define field (academics) Outline areas of future research

(acad/policy) Grant applications (academics)

4

2. Brief overview of the systematic review process

A walk through the ‘recipe’ followed for systematic reviews

5

6

FLOWCHART OF STUDIES

Merge databases (n = 7294)AND

REMOVE DUPLICATES = 5067

Inclusion Criteria Applied to

Title and Abstracts/ArticlesExcluded at this level

n = 4256

Studies considered for QA (qualitative, quantitative and mixed methods

(n =609 )

Supplemental articles from reference list

n = 63

Step 6: Finding Synthesis

Excluded at this level n = 13

Excluded at this level n = 20

n = 19

Studies passing quality appraisal n = 14

Studies passing quality appraisal n = 5

Quantitative Studies

Total considered for QA = 44

Qualitative Studies

Total considered for QA = 33

Qualitative Studies

Total considered for DE = 20

Quantitative Studies

Total considered for DE = 23

Medline

Total = 1620

EMBASE

Total = 1683

CINAHL

Total = 443

PsycINFO

Total = 1174

Sociological Abstracts Total = 279

ASSIA

Total = 74

ABI Inform

Total = 381

EconLit

Total = 58

Business Source Premier

Total = 585

Other

Total = 240

Content Experts’ Refs Total = 840

What databases will be searched?

What keywords are used in the search?

What is the scope of the search?

Languages? Dates?

What stakeholders are involved and at what stages?

How will we find papers not listed on

databases?What articles are included/excluded

?

Quality control process for what is

being included/excluded

How will we assess quality?

Two reviewers per paper, consensus

approach

What data will we extract?

Two reviewers per paper at this stage, consensus

approach

Step 1: Library Search

Step 2: Study Relevance

Step 3: Division of QNT/QL Studies

Step 4: Quality Appraisal

Step 5: Data Extraction

3. SR ‘cases’

Case I – Making decisions about how to construct systematic reviews: the early daysInterprofessional education (Scott)

Case II – Struggles with the paradigm of systematic reviews: later daysWork & health in small businesses (Ellen)

8

Interprofessional education (IPE)

Case I: Making decisions about how

to construct systematic reviews:

the early days

9

REVIEW QUESTIONwhat kind of IPE, under what circumstances, produces what kind of outcomes?

SETTING PARAMETERSDiscussion/consultation/agreement: definitions, approaches, processes

LITERATURE SEARCHESRetrieval & screening

Quantitative studies:Quality assessment, data

extraction, synthesis

SYNTHESISDifferent types of IPE and associated outcomes10

Mixed method studies:Quality assessment, data

extraction, synthesis

Qualitative studies:Quality assessment, data

extraction, synthesis

Context information

Policymaker demands for IPE evidence (early 1990s)

SR – political response In dark about SR process – new activity ‘Side line’ work (enthusiasm)

11

SR processes

Inclusive approach to review team – 9 members

Conceptually inclusive (implicit/explicit IPE) Methodologically inclusive (qual/quan) Open stance – seeking guidelines/standards Pragmatic – trial & error Abstraction – pre-determined categories

12

Thinking about Quality

quality of study (appropriate design, sampling, recruitment, validity, reliability)

quality of information (good contextual info, explicit rationale, clear research questions, clear results)

(e.g. CASP, EPPI Centre, Popay et al 1998)

13

Large and small SRs

Big was good: 107 studies Practitioners/policymakers - larger numbers (width)

Small was better - 21 studiesResearchers – small numbers (higher quality)

14

Some reflections

Lots of discussion/debate create parameters – first IPE review

Proceed with caution One step forward, two back Gradual movement: from inclusion to

exclusion

15

Some reflections

Different types of qualitative research difficult to synthesize

Qual / Quan / mixed methods – more problems

Quality assessment – best effort (pragmatics)

SR team dynamics

16

Case II: Systematic review on work and health in small businesses (Ellen)

Struggles with the paradigm of systematic reviews

2 parts: qualitative review mixed method review

17

SUB-QUESTIONSQualitative literature:

How do SB workplace parties understand and enact processes

related to OHS?

Quantitative literature:Do OHS interventions in SBs

have an effect on OHS?

REVIEW QUESTIONWhat understandings, processes and interventions influence OHS in SBs?

LITERATURE SEARCHRetrieval, screening of T&As

FOCI FOR IN-DEPTH REVIEW

Qualitative studies:Quality assessment, data

extraction, synthesis

Quantitative studies:Quality assessment, data

extraction, synthesis

OVERALL SYNTHESISUnderstandings, processes and interventions that influence OHS in SBs

STAKEHOLDER CONSULTATION

STAKEHOLDER CONSULTATION

18

Struggles in a Qualitative review

How to observe the broad focus that can be taken with qualitative research while also subjecting it to the ‘recipe’ of a SR process

The SR process is systematic, not exploratory.

Pre-defined parameters

19

Struggles in a Qualitative review

How to synthesize the findings & insights of Qualitative studies while also preserving their context and theory

The systematic review ‘extracts’ findings to answer a specific question.

What insights do we end up with? What do we gain/lose?

20

Struggles in a Qualitative review

What is being counted/considered in a qualitative systematic review?

Varying approaches to data extraction. pre-set-findings categories? grounded theory process?

21

Struggles in a Qualitative review What counts as a qualitative

study? Studies can self-identify as “qualitative”

but not use recognizable qualitative methods. E.g. Participatory studies

Our criteria: Some qualitative data & some qualitative analysis of it

22

Struggles in a Qualitative review How to handle interesting data

from studies that don’t meet quality criteria? “Nuggets” Pawson (2006). Descriptions of process that are a ‘nugget

of gold’ for the question being asked.

23

Struggles in a Mixed Method review

How to achieve a steady process for both Qualitative & Quantitative sides of the review

Quantitative team-- focused on definitions, outcomes

Qualitative team--no fixed categories for outcomes.

Qual & quan work happens at different times--affects possibilities for concurrent synergy.

24

Struggles in a Mixed Method review

Theoretical differences about ‘conflict of interest’ during the quality assessment process

Qualitative team--sensitive to social/power relations heated issue, agreed to disagree.

25

Struggles in a Mixed Method review

Differences about ‘levels of evidence’

QN additive approach E.g. "best evidence synthesis guidelines”. “How much evidence is there that this has an

effect?” QL studies—not about effectiveness

Some additive approach; e.g. concepts in 3 studies = theme

No overall conclusions about strength of evidence

26

IV. Reflections on the conduct and usefulness of systematic

reviews

SR process developed from Quantitative paradigm; we are trying to adapt Qualitative literature to it…

…Can this be meaningfully done?

27

Issues

Reviews are not entirely systematic Messiness Paired reviewers: anti-bias ‘ideal’ Politics of agreement/consensus

BUT: Power relations among reviewers Assessing papers: own merits or against

overall standard? QA approaches change over course of the

review

28

Issues

The issue of the podium Arrogance: decisions on inclusion of peer

reviewed papers

A side effect of systematic reviews Claims of ‘ownership’ of a field from SR work

29

Issues

Debates: Can the synthesis integrate research using

different methods/theories

Although there are multiple descriptions/ explanations of data, these all ultimately relate to some underlying reality/truth

(Bondas & Hall 2007, Mays et al 2005)

30

Some gains

Think carefully about papers Learn a lot about a field Read a lot of poor papers (learn: good,

bad ugly) Hone critical appraisal skills (teaching,

journal reviewing/editing) Opportunities to engage with people

(outside academia)

31

Thank you

Questions? Comments?32