PlanPlan
• GRADE backgroundGRADE background
• two stepstwo steps– quality of evidencequality of evidence– strength of recommendationstrength of recommendation
• evidence profilesevidence profiles
• an exercise in applying GRADEan exercise in applying GRADE
Why Grade Why Grade Recommendations?Recommendations?
• strong recommendationsstrong recommendations– strong methods strong methods – large precise effect large precise effect – few down sides of therapyfew down sides of therapy
• weak recommendationsweak recommendations– weak methodsweak methods– imprecise estimateimprecise estimate– small effectsmall effect– substantial down sidessubstantial down sides
Which grading system to Which grading system to use?use?
• many availablemany available– Australian National and MRCAustralian National and MRC– Oxford Center for Evidence-based Oxford Center for Evidence-based
MedicineMedicine– Scottish Intercollegiate Guidelines (SIGN)Scottish Intercollegiate Guidelines (SIGN)– US Preventative Services Task ForceUS Preventative Services Task Force– American professional organizationsAmerican professional organizations
• AHA/ACC, ACCP, AAP, Endocrine society, etc....AHA/ACC, ACCP, AAP, Endocrine society, etc....
• cause of confusion, dismaycause of confusion, dismay
A common international A common international grading system?grading system?
• GRADE (GRADE (GGrades of rades of rrecommendation, ecommendation, aassessment, ssessment, ddevelopment and evelopment and eevaluation)valuation)
• international groupinternational group– Australian NMRC, SIGN, USPSTF, WHO, NICE, Australian NMRC, SIGN, USPSTF, WHO, NICE,
Oxford CEBM, CDC, CCOxford CEBM, CDC, CC
• ~ 25 meetings over last ten years~ 25 meetings over last ten years• (~10 – 50 attendants)(~10 – 50 attendants)
GRADE UptakeGRADE UptakeAgencia sanitaria regionale, Bologna, Italia Agency for Health Care Research and Quality (AHRQ)Allergic Rhinitis and Group - Independent Expert PanelAmerican Association for the study of liver diseasesAmerican College of Cardiology FoundationAmerican College of Chest PhysiciansAmerican College of Emergency PhysiciansAmerican College of PhysiciansAmerican Endocrine Society American Society of Gastrointestinal EndoscopyAmerican society of Interventional Pain PhysiciansAmerican Thoracic Society (ATS)BMJ Clinical Evidence British Medical Journal Canadian Agency for Drugs and Technology in HealthCanadian Cardiovascular SocietyCanadian Task Force on Preventive Health CareCenters for Disease ControlCochrane Collaboration EBM Guidelines Finland Emergency Medical Services for Children National
Resource Center European Association for the Study of the LiverEuropean Respiratory SocietyEuropean Society of Thoracic SurgeonsEvidence-based Nursing Sudtirol, Alta Adiga, ItalyFinnish Office of Health Technology Assessment
German Agency for Quality in MedicineHeelth Inspectorate for ScotlandInfectious Disease Society of America Japanese Society of Oral and Maxillofacial Radiology Joslin Diabetes CenterJournal of Infection in Developing CountriesKaiser PermanenteKidney Disease International Guidelines Organization National and Gulf Centre for Evidence-based MedicineNational Institute for Clinical Excellence (NICE)National Kidney FoundationNorwegian Knowledge Centre for the Health ServicesOntario MOH Medical Advisory SecretariatPanama and Costa Rica National Clinical Guidelines Program
Polish Institute for EBMScottish Intercollegiate Guideline Network (SIGN)Society of Critical Care MedicineSociety of Pediatric Endocrinology Society of Vascular SurgerySpanish Society of Family Practice (SEMFYC) Stop TB Diagnostic Working GroupSurviving sepsis campaign Swedish Council on Technology Assessment in Health CareSwedish National Board of Health and Welfare University of Pennsylvania Health System for EB Practice UpToDate WINFOCUSWorld Allergy OrganizationWorld Health Organization (WHO)
What are we grading?What are we grading?
• two componentstwo components
• quality of body of evidencequality of body of evidence– extent to which confidence in estimate extent to which confidence in estimate
of effect adequate to support decisionof effect adequate to support decision• high, moderate, low, very lowhigh, moderate, low, very low
• strength of recommendationstrength of recommendation• strong and weakstrong and weak
Determinants of qualityDeterminants of quality
• RCTs start highRCTs start high
• observational studies start low observational studies start low
• limitations can lower quality?limitations can lower quality?
Determinants of qualityDeterminants of quality• 5 limitations can lower quality5 limitations can lower quality
• risk of biasrisk of bias– concealment, blinding, loss to follow-upconcealment, blinding, loss to follow-up
• imprecisionimprecision
• inconsistencyinconsistency– variability in results (heterogeneity)variability in results (heterogeneity)
• publication biaspublication bias
Quality judgments: Quality judgments: DirectnessDirectness
• populations populations – older, sicker or more co-morbidityolder, sicker or more co-morbidity
• interventions interventions – new statins versus oldnew statins versus old
• outcomes outcomes – important versus surrogate outcomesimportant versus surrogate outcomes– glucose control versus CV eventsglucose control versus CV events
Alendronate
Risedronate
Placebo
DirectnessDirectness
interested in A versus B interested in A versus B available data A vs C, B vs Cavailable data A vs C, B vs C
What can raise quality?What can raise quality?
• large magnitude can upgrade one levellarge magnitude can upgrade one level– very large two levelsvery large two levels
• common criteriacommon criteria– everyone used to do badlyeveryone used to do badly– almost everyone does wellalmost everyone does well– quick actionquick action
• hip replacement for hip osteoarthritiship replacement for hip osteoarthritis
• mechanical ventilation in respiratory failuremechanical ventilation in respiratory failure
Quality assessment criteriaQuality assessment criteria
Quality Assessment
Summary of Findings
QualityRelative Effect
(95% CI)
Absolute risk difference
OutcomeNumber of
participants(studies)
Risk of Bias
Consistency Directness PrecisionReportin
g Bias
Myocardial infarction
10,125(9)
No serious limitations
No serious imitations
No serious limitations
No serious limitations
Not detected
High0.71
(0.57 to 0.86)1.5% fewer
(0.7% fewer to 2.1% fewer)
Mortality10,205
(7)No serious limitations
Possiblly inconsistent
No serious limitations
ImpreciseNot
detectedModerate
or low1.23
(0.98 – 1.55)
0.5% more(0.1% fewer
to 1.3% more)
Stroke10,889
(5)No serious limitaions
No serious limitations
No serious limitations
Possible imprecision
Not detected
Moderate or High
2.21(1.37 – 3.55)
0.5% more (0.2% more to
1.3% more0
Beta blockers in non-cardiac surgery
Strength of RecommendationStrength of Recommendation
• strong recommendationstrong recommendation– benefits clearly outweigh risks/hassle/costbenefits clearly outweigh risks/hassle/cost– risk/hassle/cost clearly outweighs benefitrisk/hassle/cost clearly outweighs benefit
• what can downgrade strength?what can downgrade strength?
• low quality evidence low quality evidence
• close balance between up and downsidesclose balance between up and downsides
Risk/Benefit tradeoffRisk/Benefit tradeoff
• aspirin after myocardial infarctionaspirin after myocardial infarction– 25% reduction in relative risk 25% reduction in relative risk – side effects minimal, cost minimalside effects minimal, cost minimal– benefit obviously much greater than benefit obviously much greater than
risk/costrisk/cost
• warfarin in low risk atrial fibrillationwarfarin in low risk atrial fibrillation– warfarin reduces stroke vs ASA by 50%warfarin reduces stroke vs ASA by 50%– but if risk only 1% per year, ARR 0.5%but if risk only 1% per year, ARR 0.5%– increased bleeds by 1% per yearincreased bleeds by 1% per year
Strength of Strength of RecommendationsRecommendations
Aspirin after MI – do itAspirin after MI – do it
Warfarin rather than ASA in Afib Warfarin rather than ASA in Afib -- probably do it-- probably do it
-- probably don’t do it-- probably don’t do it
Significance of strong vs Significance of strong vs weakweak
• variability in patient preferencevariability in patient preference– strong, almost all same choice (> 90%)strong, almost all same choice (> 90%)– weak, choice varies appreciablyweak, choice varies appreciably
• interaction with patientinteraction with patient– strong, just inform patientstrong, just inform patient– weak, ensure choice reflects valuesweak, ensure choice reflects values
• use of decision aiduse of decision aid– strong, don’t botherstrong, don’t bother– weak, use the aidweak, use the aid
• quality of care criterionquality of care criterion– strong, considerstrong, consider– weak, don’t considerweak, don’t consider
Flavanoids for Flavanoids for HemorrhoidsHemorrhoids
• venotonic agentsvenotonic agents– mechanism unclear, increase venous returnmechanism unclear, increase venous return
• popularitypopularity– 90 venotonics commercialized in France90 venotonics commercialized in France– none in Sweden and Norwaynone in Sweden and Norway– France 70% of world marketFrance 70% of world market
• possibilitiespossibilities– French misguidedFrench misguided– rest of world missing outrest of world missing out
Systematic ReviewSystematic Review
• 14 trials, 1432 patients14 trials, 1432 patients
• key outcomekey outcome– risk not improving/persistent symptomsrisk not improving/persistent symptoms– 11 studies, 1002 patients, 375 events11 studies, 1002 patients, 375 events
– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57
• minimal side effectsminimal side effects
• is France right?is France right?
• what is the quality of evidence?what is the quality of evidence?
What can lower quality?What can lower quality?
• risk of biasrisk of bias– lack of detail re concealmentlack of detail re concealment– questionnaires not validatedquestionnaires not validated
• indirectness – no problemindirectness – no problem
• inconsistency, need to look at the inconsistency, need to look at the resultsresults
Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement
Study RR (random) Weight RR (random)or sub-category log[RR] (SE) 95% CI % 95% CI
01 Up to seven daysChauvenet -0.8916 (0.2376) 12.67 0.41 [0.26, 0.65] Cospite -2.2073 (0.6117) 5.51 0.11 [0.03, 0.36] Thanapongsathorn -0.4308 (0.2985) 11.18 0.65 [0.36, 1.17]
Subtotal (95% CI) 29.36 0.37 [0.18, 0.77]Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1%Test for overall effect: Z = 2.67 (P = 0.008)
02 Up to four w eeksAnnoni F -1.6094 (0.7073) 4.50 0.20 [0.05, 0.80] Clyne MB -0.9943 (0.3983) 8.94 0.37 [0.17, 0.81] Pirard J -1.1712 (0.3086) 10.94 0.31 [0.17, 0.57] Thanapongsathorn -1.1087 (1.1098) 2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan -0.8916 (0.3691) 9.56 0.41 [0.20, 0.85] Wijayanegara -0.5978 (0.1375) 14.97 0.55 [0.42, 0.72]
Subtotal (95% CI) 61.54 0.48 [0.32, 0.72]Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7%Test for overall effect: Z = 3.57 (P = 0.0004)
03 Further than four w eeksGodeberg -1.7719 (0.3906) 9.10 0.17 [0.08, 0.37]
Subtotal (95% CI) 9.10 0.17 [0.08, 0.37]Test for heterogeneity: not applicableTest for overall effect: Z = 4.54 (P < 0.00001)
Total (95% CI) 100.00 0.40 [0.29, 0.57]Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1%Test for overall effect: Z = 5.14 (P < 0.00001)
0.001 0.01 0.1 1 10 100 1000
Favours treatment Favours control
Publication bias?Publication bias?
• size of studiessize of studies– 40 to 234 patients, most around 10040 to 234 patients, most around 100
• all industry sponsoredall industry sponsored
Review : Phlebotonics for hemorrhoidsComparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement
0.001 0.01 0.1 1 10 100 1000
0.0
0.4
0.8
1.2
1.6
RR (fixed)
What can lower quality?What can lower quality?
• detailed design and executiondetailed design and execution– lack of detail re concealmentlack of detail re concealment– questionnaires not validatedquestionnaires not validated
• inconsistencyinconsistency– almost all show positive effect, trendalmost all show positive effect, trend– heterogeneity p < 0.001; Iheterogeneity p < 0.001; I22 65.1% 65.1%
• indirectnessindirectness
• imprecisionimprecision
– RR 0.4, 95% CI 0.29 to 0.57RR 0.4, 95% CI 0.29 to 0.57
• reporting biasreporting bias– 40 to 234 patients, most around 10040 to 234 patients, most around 100
Is France right?Is France right?
• recommendationrecommendation– yesyes
– no against useno against use
• strengthstrength– strong strong – weakweak
ConclusionConclusion
• clinicians, policy makers need summariesclinicians, policy makers need summaries– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations
• explicit rulesexplicit rules– transparent, informativetransparent, informative
• GRADEGRADE– simple, transparent, systematicsimple, transparent, systematic– increasing wide adoptionincreasing wide adoption