Evidence to Decision tables – 11th CE&B's research day presentation

Case study

From evidence to decisions

Frameworks for Clinical Guidelines’ Development

Carlos A. Cuello, MD, PhD(c) Health Research Methodology Program!Department of Clinical Epidemiology and Biostatistics

March 27, 2014

Disclosure

Authors belong to the CE&B department and

to the GRADE / DECIDE working group.

No other conflicts to declare.

Usefulness of the Evidence to DECISION table

for clinical practice guidelines: a case study

with a panel of international experts

Carlos Cuello-Garcia, Jan Brozek, Juan José Yepes-Nuñez, Yuan Zhang, Shreyas Gandhi, Arnav Agarwal, Holger Schünemann

Clinical Epidemiology and Biostatistics McMaster University MacGRADE Center

What are these frameworks?

How can they help?

Why should we use them?

decision

(recommendation)

research

evidence

benefits vs

harms

resource use

patient

values

equity

feasible?

acceptable?

Clinical Guideline

FRAMEWORKS

evidence to decision (EtD)

just 4 columns

& conclusions

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

JUDGEMENTSRESEARCH EVIDENCE

ADDITIONAL INFORMATION



��

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility



��

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

��

��



CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

��



��

CRITERIA

Problem

Quality of evidence

Benefits & harms

Values

Resource use

Equity

Acceptability

Feasibility

CONCLUSIONS

Balance of consequences

Decision /recommendation

Justification

Implementation considerations

Monitoring

Evaluation

Research priorities

�� …

PURPOSE OF THE FRAMEWORK

PROS CONS

Inform decision makers’ judgements about the pros and

cons of each option (intervention) that is considered.

The important factors that determine a decision (criteria) are considered.

Structure discussion and manage disagreements

Make the basis for decisions transparent to guideline users

Provide a concise summary of the best available research evidence to inform judgements about each

criterion

A case study . . .

Guideline for Allergic Diseases Prevention

INTERNATIONAL PANEL

guidelines on allergy prevention

• Probiotics

• Prebiotics

• Vitamin D

guidelines on allergy prevention

• Probiotics

• Prebiotics

• Vitamin D

Explanations Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013

Generic EtD framework 1

Evidence to decision framework

Question 1: Should probiotics vs. no probiotics be used in pregnant women?

Population:!pregnant!women!

Option:!probiotics!Comparison:!no!probiotics!Setting:3outpatient!Perspective:3individual!patient

Background: The$intestinal$microbiome$could$play$an$important$role$in$the$immune$system$maturation,$and$it$has$been$suggested$that$early6life$probiotic$administration,$whether$directly$to$the$infant$or$in$their$mothers$breast$milk,$may$reduce$the$risk$of$allergies$in$childhood.$The$objective$of$this$question$is$to$evaluate$the$impact$of$probiotics$administered$to$the$expecting$mothers$on$their$infant.

Subgroup considerations: subpopulation of women at high risk for allergy in a child

CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION

PR

OB

LEM

Is the problem a priority?

No Probably no Uncertain Probably yes Yes

X

Allergic diseases represent a spectrum of health conditions and a worldwide burden in different populations. (1)

Are a large number of people affected?

No Probably no Uncertain Probably yes Yes

X

As many as 40% of the worldwide population is affected by any type of allergy. In infants prevalence depends highly on the allergic status of their parents, being approximately of 10% in those without an allergic parent or sibling, versus 20% to 30% in those with the atopic background in their relatives. (2)

Prepared by: Holger, Jan, Carlos, Juan Date: December 10, 2013



CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS

VA

LUE

S

Is there important uncertainty or variability about how much people value the main outcomes?

Important uncertainty

or variability

Possibly important

uncertainty or

variability

Probably no

important uncertainty

or variability

No important

uncertainty or

variability

No known undesirable outcomes

X

Detailed judgements

The relative importance or values of the main outcomes of interest:

Outcome Relative importance Certainty of the evidence Eczema critical low

Asthma/wheezing critical low

Food allergy critical low

Adverse effects critical low

We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk.

BE

NE

FIT

S &

HA

RM

S O

F T

HE

OP

TIO

NS

What is the overall certainty of the evidence of effectiveness?

No included studies Very low Low Moderate High

X

Summary of findings:

Outcome With [intervention]

Without [intervention]

Difference (per 100) (95%CI)

Relative effect (RR)

(95%CI)

Certainty of the

evidence (GRADE)

Eczema (follow-up 1 to 5 years)

365/1520 (24%)

484/1515 (31.9%)

9 fewer per 100 (from 4 fewer to 13

fewer)

RR 0.72 (0.6 to 0.86)

⊕⊝⊝⊝ VERY LOW

Asthma/wheezing (follow-up 2 to 7 years)

143/992 (14.4%)

139/982 (14.2%)


more)

RR 0.97 (0.77 to 1.22)

⊕⊕⊝⊝ LOW

Food allergy (follow-up 1 to 2 years)

36/279 (12.9%)

41/284 (14.4%)

1 more per 100 (from 3 fewer to 8

more)

RR 1.08 (0.73 to 1.59)


Adverse effects 101/394 (25.6%)

88/397 (22.2%)


more)

RR 1.13 (0.82 to 1.52)


Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups

The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy.

How substantial are the desirable anticipated effects?

Don’t know

Not important

Somewhat important

Moderately important

Very important

Varies

X

X

Detailed judgements

There was some disagreement among panel members whether the effect is somewhat or moderately important.

How substantial are the undesirable anticipated effects?

Don’t know

Very important


Somewhat important

Not important

Varies

X

Detailed judgements

No serious adverse effects, and no difference in mild adverse effects between the groups.

Do the desirable effects outweigh the undesirable effects?

No Probably No

Don’t know Probably Yes

Yes Varies

X

Detailed judgements





VA

LUE

S

Is there important uncertainty or variability about how much people value the main outcomes?

Important uncertainty

or variability

Possibly important

uncertainty or

variability

Probably no

important uncertainty

or variability

No important

uncertainty or

variability

No known undesirable outcomes

X

Detailed judgements

The relative importance or values of the main outcomes of interest:

Outcome Relative importance Certainty of the evidence Eczema critical low

Asthma/wheezing critical low

Food allergy critical low

Adverse effects critical low

We judged that the outcomes eczema, asthma and food allergy are critical for people. The adverse outcomes are probably of high importance and the burden of taking daily pills is limited. Some immunocompromised women might not accept the risk.

BE

NE

FIT

S &

HA

RM

S O

F T

HE

OP

TIO

NS

What is the overall certainty of the evidence of effectiveness?

No included studies Very low Low Moderate High

X

Summary of findings:

Outcome With [intervention]

Without [intervention]

Difference (per 100) (95%CI)

Relative effect (RR)

(95%CI)

Certainty of the

evidence (GRADE)

Eczema (follow-up 1 to 5 years)

365/1520 (24%)

484/1515 (31.9%)


fewer)

RR 0.72 (0.6 to 0.86)


Asthma/wheezing (follow-up 2 to 7 years)

143/992 (14.4%)

139/982 (14.2%)


more)

RR 0.97 (0.77 to 1.22)

⊕⊕⊝⊝ LOW

Food allergy (follow-up 1 to 2 years)

36/279 (12.9%)

41/284 (14.4%)


more)

RR 1.08 (0.73 to 1.59)


Adverse effects 101/394 (25.6%)

88/397 (22.2%)


more)

RR 1.13 (0.82 to 1.52)


Link to detailed evidence profile Subgroup considerations: Link(s) to summary of findings and judgments for subgroups

The data are indirect for all outcomes because they are primarily derived from studies that looked at mixed exposure in women during pregnancy and breastfeeding and of infants after birth. Only 1 RCT assessed the effect on eczema in pregnant women only: RR 0.88 (0.63 to 1.22); RD 5 fewer per 100 (from 14 fewer to 9 more) 5 RCTs included pregnant women + later breastfeeding mothers: RR 0.5 (0.4 to 0.63); RD 21 fewer per 100 (from 15 fewer to 25 fewer) 5 RCTs included pregnant women + infants after birth (follow-up 1 to 5 years): RR 0.87 (0.72 to 1.04), RD 4 fewer per 100 (from 8 fewer to 1 more) 3 RCTs included pregnant women + subsequently breastfeeding + infants (follow-up 3 to 4 years): RR 0.78 (0.49 to 1.24); RD 7 fewer per 100 (from 17 fewer to 8 more) No effects were observed on asthma/wheezing and food allergy.

How substantial are the desirable anticipated effects?

Don’t know

Not important

Somewhat important


Very important

Varies

X

X

Detailed judgements

There was some disagreement among panel members whether the effect is somewhat or moderately important.

How substantial are the undesirable anticipated effects?

Don’t know

Very important


Somewhat important

Not important

Varies

X

Detailed judgements

No serious adverse effects, and no difference in mild adverse effects between the groups.

Do the desirable effects outweigh the undesirable effects?

No Probably No

Don’t know Probably Yes

Yes Varies

X

Detailed judgements





RE

SO

UR

CE

US

E

How large are the resource requirements?

Large costs

Moderate costs

Small Moderate savings

Large savings

Varies

X

Detailed judgements

Prices are likely to vary substantially depending on the setting. This may be a particularly important consideration in low and middle-income countries. A level and type of insurance may play a substantial role as well. From a health systems point of view it might also be cost effective given that probiotic would be used for 9 months and cost of treatment of eczema may be distributed across many years.

Extremely limited research evidence (internet searches of drug prices)

Bifidobacterium bifidum (cost per person per year US$) Dose: 1 pill

each day

Lactobacillus gg (cost per person per

year US$) 1 pill each day

North-America

Average $181.16 $341.6

South-America

Average $174.3 $286

Europe

Average $167.86 $251.56

Fewer office visits would occur as a result of eczema if the effects on eczema were true.

How large is the incremental cost relative to the net benefit?

Very large ICER

Large ICER

Moderate ICER

Small ICER

Savings Varies

Detailed judgements

No research evidence

If eczema was reduced the intervention might be cost-effective given fewer office visits (between $17,400 and $34,100 to treat 100 people for 1 year or ¾ of that for 9 months) preventing 9 cases of eczema. In most studies probiotics were used in the last trimester of pregnancy, which, if used this same way, might reduce the cost per pregnant woman.

EQ

UIT

Y What would

be the impact on health inequities?

Increased Probably increased

Uncertain Probably reduced

Reduced Varies

X

Detailed judgements


In some settings it may be important to consider equity as the access may depend on socioeconomic status of the country or setting where coverage will depend on policymakers.

AC

CE

PT

AB

ILIT

Y

Is the option acceptable to key stakeholders?

No Probably No

Uncertain Probably Yes

Yes Varies

X

Detailed judgements






FE

AS

IBIL

ITY

Is the option feasible to implement?

No Probably No

Uncertain Probably Yes

Yes Varies

X

Detailed judgements





Recommendation Should probiotics vs. no probiotics be used in pregnant women (exposing their children in utero)?

Overall balance of consequences Undesirable consequences clearly outweigh desirable

consequences

Undesirable consequences probably outweigh desirable

consequences

The balance of desirable and undesirable consequences indicates

they are very similar*

Desirable consequences probably outweigh undesirable

consequences

Desirable consequences

clearly outweigh undesirable

consequences

! ! ! X !

We recommend against the option or for the alternative

We suggest not to use the option or to use the alternative

We suggest using the option We recommend the option

! ! X !

Panel decisions 3 panel members with potential COI recused themselves from participating in formulating the recommendation. Consensus was obtained from the rest of the team.

Recommendation (text) The guideline panel suggests using probiotics in pregnant women at high risk for allergy in their children (conditional recommendation, very low quality evidence).

Remarks and justification Most studies commenced probiotics in the last trimester of pregnancy. The very low quality evidence for adverse effects indicates that our confidence in the absence of increased adverse effects is low. Future research is needed (see definitions of very low quality) e.g., generalizing to immune-compromised children

Subgroup considerations Women with high risk of allergy in their children Women with average risk of allergy in their children

Implementation considerations This recommendation is based on trials investigating the probiotics or mixtures of probiotic listed below. We have not found a difference between these different probiotics, but that does not mean there is no difference

Monitoring and evaluation considerations

Research priorities Develop instruments for evaluating the risk of allergy in children as the family history predicts only about 30% of the population risk. There is some evidence that first child is at higher risk for allergy than subsequent children. Long-term follow-up of long-term effects. No direct evidence for the use of probiotics in formula – this should be evaluated in future research and is an unmet need.


SURVEY

24 panel members

13 RESPONDED

11 ACCEPTED TO COMPLETE THE

SURVEY

1. The EtD table helped in the ORGANIZATION of the development process (i.e., for getting a recommendation from a body of evidence)

Strongly disagree

DisagreeSomewhat disagree

neutralSomewhat

agreeAgree Strongly agree

2. The EtD is a structured process that is BETTER than an informal/unstructured discussion

Strongly disagree


neutralSomewhat


3. The EtD helped you UNDERSTAND the process of getting a recommendation from a body of evidence

Strongly disagree


neutralSomewhat


4. The EtD made you think about concepts like patient values & preferences, costs & risks of the intervention, equity, acceptance, and feasibility of the intervention

Strongly disagree


neutralSomewhat


– Anonymous panel member

“It made the process smooth, easy to understand and significantly reduced the time it would have otherwise taken to complete”

What are these frameworks?

How can they help?

Why should we use them?

Gracias@CharlieNeck

[email protected]

Health & Medicine

Evidence to Decision tables – 11th CE&B's research day presentation