Issues and challenges to scoping and focusing the question

ESQUIRE Qualitative Systematic Review Workshop University of Sheffield8 September 2011

Janet HarrisCochrane Qualitative Research Methods Group

Session aims

• To identify the issues and challenges in formulating different types of qualitative questions for qualitative evidence synthesis

The issue: What question?• The challenges

•Utility•Relevance•Timeliness•Explanatory value

How effectiveness questions are developed

Refine definition of

problem

Identify possible

causes and effects

Identify problem

Test solution

s

Margerunm-Leys http://www-personal.umich.edu/~jmargeru/conceptmap/types.htm

Clinical expertise + research evidence are used to:

Effectiveness question: In adults with Low Back Pain does provision of individual education reduce pain, improve functioning and promote return to work?

P: Adults with back pain (BP)

O: Reduce pain, improve

functioning, return to work

I: Information on how to self-manage BP

Assumed

solution:Patients

need written and/or verbal

education delivered

by a health professiona

l

Problem and

solution (as defined

by the researcher

s)

Possible results from a search using an informing question

Problem Definition(s)

Effects

CausesInforming question

(Background question)

Single studies / evidence syntheses identifying factors that might make us rethink assumptions

Explore assumptions with informing questions

P: Important characteristics

relating to I and O

O: Factors that may dilute

potential effectiveness

I: Factors that affect

implementation

Informing question

(Background question)

Refine PConsider subgroup analysis

Consistency of

implementation

Such as participants’

views

Back pain patients’ experiences of receiving education for self management

Informing questions may identify theories• Theories are explanations for a phenomenon or

an event e.g. Why something happens in relation to another event

• They can be grand theories, they can be middle range theories, or they can be single-case theories

• Middle range theory goes beyond individual studies, providing a cross cutting explanation for patterns that are seen across settings and different types of people

• Whenever possible, interventions testing effectiveness should be based on theory

Issues raised from informing questions

• There may be ‘essential ingredients’ that need to be included in interventions• For example, the search on back pain education

identified an issue with appropriateness and relevance of information

• Informing questions need to produce convincing evidence about these ingredients to justify changing an effectiveness question.

• How much evidence is needed to change the original effectiveness question?

Revising the effectiveness question: An example

• In adults with Back Pain does provision of individual education by a health professional reduce pain, improve functioning and promote return to work?

• Revised effectiveness question:• P = adults with Low Back Pain • I = information explained by health professional• C= information explained by a health

professional with additional peer support• O = reduce pain, improve functioning and

promote return to work

Additional ingredients that theoretically increase relevance

Issues with the refined question• Is the quality of reporting for effectiveness studies

adequate?• For example, do the interventions describe how

information is explained by the health professionals? • Is the amount of time for the explanation documented? • Does the comparison describe the type of peer support

offered?

• Qualitative evidence can indicate how a question should be refined, but if effectiveness studies do not report the relevant information then the end product could be an empty review

Enhancing questions to explain the results of effectiveness reviews

• Monitors pathway

from interventi

on to outcomes

• May highlight problems

with Quan

studies

Enhancing question• Within-study data

• Parallel-study data

• Similar-study data

Enhancing questions: Issues and challenges

• Adequate data within the study about PICO characteristics?

• Adequate data in embedded qualitative studies?• Adequate data in parallel sibling studies?• Can ‘similar’ qualitative studies be found?• What does ‘similar’ mean – across embedded, sibling,

and independently conducted studies? • Comparability of data - how to synthesize data across

studies?

Utility of findings: Used to critique

design and PICO in

included studies;

subgroup analysis of

theory-based interventions

..

Extending question: Can explore an issue related to population, intervention, comparison or outcome

Cause: Low awareness of educational

approaches for behaviour change

Extending question:How do people make sense of and use Back Pain information? What educational theories promote better use of BP information?

Problem noted: Lack of theory-based education on how to self-

manage BP

Negative effect: Increased

utilization of services and absenteeism

Extending questions: Issues and challenges• Extending questions require a separate qualitative

evidence synthesis, either conducted within or alongside an effectiveness review. Funding needs to be justified by considering:• Timeliness: Is the information perceived to be critical to

using the results of the effectiveness review? • Relevance: Is the issue important to funders, policymakers

and effectiveness reviewers ?• Utility: did the scoping process specifically note issues that

need to be informed by qualitative research?• Explanatory value: Will the qualitative synthesis contribute

to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness?

Extending questions: Issues and challenges• Extending questions require coordination of a team

including qualitative evidence synthesis (QES) members working alongside effectiveness review (ER) members

• The parallel processes need to be considered because• Findings from the QES may identify quality issues in the

effectiveness studies. At what point are these quality issues brought into the effectiveness review – at abstract review? Development of inclusion criteria? Critical appraisal? Metaanalysis?

• Reporting QES process and findings: Approaches are still being developed. At the same time, the software for Cochrane ERs – RevMan – does not yet have the facility for including Extending reviews.

Effectiveness review

Supplementing questions

Supplementing review:

Conducted independently

Before or after an ER

On a wider issue than

effectiveness

Effectiveness review

Supplementing question issues• Supplementing questions: Used in stand alone reviews

to produce information on a wider issue • Does peer support help people with chronic conditions?• Does peer support promote health literacy (help people to

understand and use health information)• Issues

• Timeliness: Will the theory be used over time?• Relevance: Does the theory address issues noted by ER

authors and those trying to implement review results (External validity)

• Utility: will the theory enable development of better effectiveness studies?

• Explanatory value: Can the theory be used to explain some of the differences in effect within and across existing ERs? Can the theory be used to inform development of future ERs?

Summary: What guides qualitative question formulation• The need to ensure that effectiveness questions are

informed by existing theory on what works (Informing questions)

• The need to provide real-time explanations for effectiveness studies (Enhancing questions)

• The need to further explore issues identified in the review by reviewing a broader range of research on effectiveness (Extending question)

• The need to explore the context surrounding a condition or an intervention, including situations and experiences that may not be directly related to effectiveness (Supplementing question)

Is it worth it?• When scoping and focusing the question, we need to ask:

Does this question pass the TRUE test?• Timely: Will the question still be relevant when the

review is finally completed? • Relevant: How will this question contribute information

that is relevant to health services policymakers?• Useful: Which of (the many) findings from the qualitative

scoping will be useful in focusing the review and/or explaining the results?

• Explanatory value: Does the synthesis contribute to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness?

Further challenges

• Team skills• Team coordination• Informed and transparent decisions

about when and how to useQES/ER data