Introduction - Cochranenetherlands.cochrane.org/sites/netherlands.cochrane... · • lack well-articulated conceptual framework • criteria not comprehensive or transparent GRADE

IntroductionTon Kuijpers

Content

• Background and rationale for revisiting guideline methodology

• GRADE approach

– Quality of evidence

– Strength of recommendations

• Grade Working Group

Systematic review

Guideline development

P

I

C

O

Outcome

Outcome

Outcome

Outcome

Formulate question

Rate importance

Critical

Important

Critical

Not important

Create

evidence profile

with GRADEpro

Summary of findings

& estimate of effect for each outcome

Rate

overall quality of evidence

across outcomes based on

lowest quality

of critical outcomes

RCT start high,

obs. data start low

1. Risk of bias

2. Inconsistency3. Indirectness

4. Imprecision

5. Publication

bias

Gra

de

do

wn

Gra

de

up

1. Large effect

2. Dose response

3. Confounders

Rate quality of

evidence for

each outcome

Select outcomes

Very low

Low

Moderate

High

Formulate recommendations:

•For or against (direction)

•Strong or weak (strength)

By considering:�Quality of evidence

�Balance benefits/harms

�Values and preferences

Revise if necessary by considering:

�Resource use (cost)

• “We recommend using…”

• “We suggest using…”• “We recommend against using…”

• “We suggest against using…”

Outcomes

across studies

Content


• GRADE approach




Different grading systems

Recommendation for use of oral anticoagulation in patients with

atrial fibrillation and rheumatic mitral valve disease

Evidence Recommendation

• B Class I

• A 1

• IV C

Organization

� AHA

� ACCP

� SIGN

Which grading system?

Hierarchy of evidence

Gradering

A1 Systematic reviews / meta-analysis

A2 Randomised controlled trials (RCT)

B Comparative studies (i.e. cohort studies)

C Non-comparative studies

D Expert opinion


Gradering

A1 Systematic reviews / meta-analysis

A2 Randomised controlled trials (RCT)

B Comparative studies (i.e. cohort studies)

C Non-comparative studies

D Expert opinion


Expert Opinion

BiasStudy design

•Randomized Controlled Trials

•Cohort Studies and Case Control Studies

•Case Reports and Case Series,

Non-systematic observations

•Expert opinion


Expert O

pinion

Bias

Limitations of existing systems

• confuse quality of evidence with strength of recommendations

• lack well-articulated conceptual framework

• criteria not comprehensive or transparent

GRADE unique

• breadth, intensity of development process

• wide endorsement and use

• conceptual framework

• comprehensive, transparent criteria

• focus on all important outcomes related to a specific question

and overall quality

Content


• GRADE approach




The GRADE approach

1. quality of evidence (4 categories)

• high

• moderate

• low

• very low

2. recommendations (2 grades)

• strong

• weak (conditional)

The GRADE approach

1. quality of evidence (4 categories)

• high

• moderate

• low

• very low

Systematic review


P

I

C

O

Outcome

Outcome

Outcome

Outcome

Formulate question

Rate importance

Critical

Important

Critical

Not important

Create

evidence profile

with GRADEpro

Summary of findings


Rate



lowest quality


RCT start high,

obs. data start low

1. Risk of bias


4. Imprecision

5. Publication

bias

Gra

de

do

wn

Gra

de

up

1. Large effect

2. Dose response

3. Confounders

Rate quality of

evidence for

each outcome

Select outcomes

Very low

Low

Moderate

High












Outcomes

across studies

I B II V III

Quality: High

Quality: Moderate

Quality: Low

Old system

GRADE is outcome-centric

Rate the relative importance of the outcomes

• Patient important outcomes!

• Maximum 7

• Rate the importance on a 9-point scale

7 – 9 critical for making a decision

4 – 6 important, but not critical for making a decision

1 – 3 not important

Rate the relative importance of the outcomes

Hierarchy of outcomes according to their importance to assess the effect of phosphate-lowering drugs in patients with renal failure and

hyperphosphatemia (Guyatt GH et al. Journal of Clinical Epidemiology 64 (2011) 395-400)

Definition ‘quality of evidence’

• The quality of evidence reflect the extent of our confidence that

the estimates of the effect are correct

Quality of evidence

Quality Symbol Interpretation

High ⊕⊕⊕⊕ We are very confident that the true effect lies close to that of the

estimate of the effect.

Moderate ⊕⊕⊕Ο We are moderately confident in the effect estimate: The true effect

is likely to be close to the estimate of the effect, but there is a

possibility that it is substantially different

Low ⊕⊕ΟΟ Our confidence in the effect estimate is limited: The true effect

may be substantially different from the estimate of the effect.

Very low ⊕ΟΟΟ We have very little confidence in the effect estimate: The true

effect is likely to be substantially different from the estimate of

effect.

Quality criteria

Design Initial quality of a body of

evidence

Lower if Higher if Quality of body of evidence

RCT High → Study limitations

Inconsistency

Indirectness

Imprecision

Publication bias

Large effect

Dose response

All plausible

confounding &

bias

−would reduce a

demonstrated effect−would suggest a

spurious effect when

results show no effect

High

Moderate

Observational

study

Low → Low

Very low

Determinants of quality

• RCTs start high

• observational studies start low

• 5 factors lower the quality of evidence

– risk of bias (Risk of Bias)

– inconsistency

– indirectness

– imprecision

– reporting bias

Determinants of quality

• 3 factors can increase the quality of evidence

– strong association

– dose-respons relationship

– all plausible confounding & bias

Overall quality of evidence


=

Quality of evidence of the critical outcome with lowest rating

GRADE evidence profile

GRADEpro

http://www.cc-ims.net/gradepro

Example: chronic non-specific low back pain

Key question

Should NSAIDS be used for chronic non-specific low back pain?

Relative importance of the outcomes

Patient relevant outcome Mean score*

1. Pain 8 CRITICAL

2. Disability 8 CRITICAL

3. Return to work 7 CRITICAL

4. Quality of life 6 IMPORTANT

5. Side effects 8 CRITICAL

* Mean score of the 14 members of the GDG (including 2 patients)

Analysis

Analysis 01.01 NSAIDs vs placebo for chronic non-specific LBP, Change in Pain Intensity from baseline on 100mm VAS.

Follow up <=12 weeks.

Analysis 01.02 NSAIDs vs placebo for chronic non-specific LBP, Total adverse events. Follow up <=12 weeks.

GRADE evidence profile


=

low

Systematic review


P

I

C

O

Outcome

Outcome

Outcome

Outcome

Formulate question

Rate importance

Critical

Important

Critical

Not important

Create

evidence profile

with GRADEpro

Summary of findings


Rate



lowest quality


RCT start high,

obs. data start low

1. Risk of bias


4. Imprecision

5. Publication

bias

Gra

de

do

wn

Gra

de

up

1. Large effect

2. Dose response

3. Confounders

Rate quality of

evidence for

each outcome

Select outcomes

Very low

Low

Moderate

High












Outcomes

across studies

Content


• GRADE approach




Evidence to recommendation (EtR)

Factors

Evidence

Factors

Recommendation

Strength of recommendation

“The strength of a recommendation reflects the extent to which we

can, across the range of patients for whom the recommendations

are intended, be confident that desirable effects of a management

strategy outweigh undesirable effects.”

Desirable and undesirable effects

• Desirable effects

– mortality

– improvement in quality of life, fewer

hospitalizations/infections

– reduction in the burden of treatment

– reduced resource expenditure

Desirable and undesirable effects

• Undesirable effects

– deleterious impact on morbidity, mortality or quality of life,

increased resource expenditure

Determinants of the strength of a recommendation

• Quality of the evidence

• Balance between desirable and undesirable effects

• Values and preferences

• Costs (resource allocation)

Developing recommendations

Implications of a strong recommendation

• Patients: Most people in this situation would want the

recommended course of action and only a small proportion

would not

• Clinicians: Most patients should receive the recommended

course of action

• Policy makers: The recommendation can be adapted as a

policy in most situations

Implications of a weak recommendation

• Patients: The majority of people in this situation would want

the recommended course of action, but many would not

• Clinicians: Be prepared to help patients to make a decision

that is consistent with their own values/decision aids and

shared decision making

• Policy makers: There is a need for substantial debate and

involvement of stakeholders

Example: chronic non-specific low back pain

Key question

Should NSAIDS be used for chronic non-specific low back pain?


=

low

Overall quality of evidence Low

Factors Judgement Explanation

1. Quality of evidence

Is there high or moderate quality evidence? ☐ yes☒ no

…

2. Balance of benifits versus harms and burdens

Are you confident that the benefits outweigh the harms and

burden or vice versa?

☐ yes☒ no

…

3. Values and preferences

Are you confident about the assumed or identified relative values

and are they similar across the target population?

☐ yes☒ no

…

4. Resource implications

Are the resources worth the expected net benefit from following

the recommendation?

☐ yes☒ no

…

Strength of recommendation Weak (conditional)

Recommendation

We suggest the use of NSAID’s for pain relieve on the short term. Due to

potential harms it important to discuss alternative options (wait and see

policy) with the patient.

Given the low quality of the (overall) evidence and the uncertainty of the balance

between benefits and harms the GDG did only suggest the use of NSAID’s for

pain relieve. It is important to discuss the potential harms and alternative

options with the patient.

Rationale

Content


• GRADE approach




GRADE Working Group

• 2x a year meeting ‘GRADE Working Group’

• Publications in -among others- BMJ (2008) and JCE (2011/2013)

Website: www.gradeworkinggroup.org

Organizations

2005 2006 2007 2008 2009 2010 201320122011

Systematic review


P

I

C

O

Outcome

Outcome

Outcome

Outcome

Formulate question

Rate importance

Critical

Important

Critical

Not important

Create

evidence profile

with GRADEpro

Summary of findings


Rate



lowest quality


RCT start high,

obs. data start low

1. Risk of bias


4. Imprecision

5. Publication

bias

Gra

de

do

wn

Gra

de

up

1. Large effect

2. Dose response

3. Confounders

Rate quality of

evidence for

each outcome

Select outcomes

Very low

Low

Moderate

High












Outcomes

across studies

Conclusions

• GRADE is gaining acceptance as international standard

• Criteria for evidence assessment across questions and outcomes

• Criteria for moving from evidence to recommendations

• Simple, transparent, systematic

– four categories of quality of evidence

– two grades for strength of recommendations

• Transparency in decision making and judgments is key

Documents

Introduction - Cochranenetherlands.cochrane.org/sites/netherlands.cochrane... · • lack well-articulated conceptual framework • criteria not comprehensive or transparent GRADE