Literature at your fingertips
May 24, 2013
Peter Tugwell
• Participants will learn how to formulate questions that can be answer by available evidence
• Participants will learn an approach to critically appraise the literature
• Participants will learn the key features of systematic reviews and how to use them to inform clinical practice
2
Learning Objectives
• None applicable.
Disclosures
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’ FIGURE
GUIDELINES GRADE AMSTAR
©
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
THE 6 A’S -STEPS OF EVIDENCE BASED PRACTICE
1. ASK a focussed question2. ACCESS - search for epidemiological evidence to help
answer question3. APPRAISE the evidence for its validity, effect size,
precision)4. AGGREGATE the evidence using the
triangle,circle,box,x FRAMEWORK5. APPLY your decision integrating the aggregated
evidence into the trade-off of (i)benefits versus harms;(ii)patient values and preferences,(iii) cost –effectiveness and cost- equity,to make an evidence-based decision
6. AUDIT your practice (i.e. check your actual practice against evidence-based practice on a regular basis)].
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. P2. I3. C4. O5. T
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. Participants (patient(s) you want to treat)
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. Participants (patient(s) you want to treat)2. Intervention( or ‘Exposure’ if no intervention )
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. Participants (patient(s) you want to treat)2. Intervention( or ‘Exposure’ if no intervention )3. Comparison (there is always an alternative! -
another therapy, nothing …
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. Participants (patient(s) you want to treat)2. Intervention( or ‘Exposure’ if no intervention )3. Comparison (there is always an alternative! -
another therapy, nothing …4. Outcome (MCID [Minimal Clinically Important Difference] in the
Humanly Important Outcome [Distress,Disability,Dysfunction,Death]
EBP STEP 1: ASK - TURN YOUR QUESTION INTO 5 PARTS (PICOT)
1. Participants (patient(s) you want to treat)2. Intervention( or ‘Exposure’ if no intervention )3. Comparison (there is always an alternative! -
another therapy, nothing …4. Outcome (MCID [Minimal Clinically Important Difference] in the
Humanly Important Outcome [Distress,Disability,Dysfunction,Death]
5. Time frame (over which you expect a result)
2012 ACR OA Scenarios-Knee
Base Case 1 (knee OA)Patients:An adult with symptomatic knee OA without co-morbidities, presents to her/his primary care provider for treatment with mild to severe pain in and/or around her knee(s) .No response to acetominophen.Intervention: NSAID Comparator : PlaceboOutcome: Pain, Function, Adverse Effects
Scenarios-Knee (con’t)
– Peptic ulcer Scenario : …she/he has a history of symptomatic and/or complicated peptic ulcer disease (e.g., history of upper GI bleed but not in the past year).
Scenarios-Knee (con’t)
– Low Dose Aspirin: …she/he is taking low-dose (325 mg/day or less) aspirin for cardioprotection
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’ FIGURE
GUIDELINES GRADE AMSTAR
©
P
I C
OT
• P
• I
• C
• O
• T
• Recruitment
• Allocation
• Maintenance
• Blind or
• Objective measurements &
processes
the evidence using RAMBO on the PICOT frame
Figure
©
Epidemiologic evidence
Clinical / population
health considerations
Policy issues
Patient / community preferences
Guidelines: making evidence-based decisions
expertise: ‘putting it all together’ the art of practice
EBP Step 4: AGGREGATE the relevant information & make an evidence-based decision:’ the X-factor
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’ FIGURE
GUIDELINES GRADE
©
OUR EXPERIENCE WITH DEVELOPING OA GUIDELINES FOR ACR
Roy Altman, Maria Benkhalti, Elizabeth Ghogumu, Gordon Guyatt, Marc Hochberg, Jessie McGowan,
Annette O’Connor, Karine Toupin-April, Peter Tugwell , Erin Ueffing, Vivian Welch, George Wells
June 1st, 2009Rheumatology city-wide rounds, Ottawa
Limitations of Previous Methods for developing Guidelines
• Methods Lack of clarity in presentation of results – Clinicians do not understand what they mean!
Ie diffference between 1a 2b 3d – Lack of validated yet simple qua 3d lity – Both Absolute and Relative effects often not reported – Absolute NNT/NNH often not calculated– Only SMDs
• Absence of clinically relevant scenarios in some
2008 Request from ACR for new OA Guidelines
To develop guidelines for the management of osteoarthritis of the hip, knee and handMethods considered
– Rand Appropriateness Panel– GRADE (Grades of Recommendation, Assessment,
Development and Evaluation) method
Proposal to ACR for OA Guidelines
Pharmacologic and non-pharmacologic interventionsUse of ‘GRADE’ to create Summary of Findings tables and to make recommendations
Use of the Wells calculator to find – The relative risk for continuous outcomes – The number needed to treat (NNT)
Use of clinical scenarios to ensure clinical relevance
Clinically relevant treatment comparisons (>50)
ACR Proposal
Pharmacologic and non-pharmacologic interventions
Knee and Hip OA TreatmentsAcupunctureTENS AcetaminophenChondroitin SulfateCortico-steroid injectionGlucosamine SulfateOpioidsTramadol
Oral NSAIDsTopical capsaicinTopical NSAIDsHyaluronates injectionOther non-pharmacologic PT and OT
– [incl weight loss, exercise, knee brace,foot insole]
> 50 comparisons
ACR Scope of work
Treatments specific to Hand OA
Oral Trolamine salicylateOral methotrexateOral hydroxychloroquineOral sulfasalazine
ACR Scope of work
ACR Proposal
Use of the new ‘GRADE’ Method
‘GRADE’ (Grades of Recommendation, Assessment, Development and Evaluation) method
– to create Summary of Findings tables and to make recommendations
GRADE system uptake (Grades of Recommendation, Assessment, Development and Evaluation)
Up-To-Date British Medical JournalACP BMJ Clinical EvidenceSociety of Vascular SurgeryEuropean Respiratory SocietySociety of Critical Care Medicine American College of Chest Physicians EBM Guidelines FinlandNational Institute for Clinical Excellence (NICE)Agency for Health Care Research and Quality (AHRQ)Swedish National Board of Health and WelfareCanadian Agency for Drugs and Technology in Health Ontario MOH Medical Advisory Secretariat
World Health OrganizationAmerican Thoracic Society Cochrane CollaborationPolish Institute for EBM Society of Pediatric EndocrinologyAmerican Endocrine SocietySurviving sepsis campaignEuropean Soc of Thoracic Surgeons Allergic Rhinitis in Asthma GuidelinesEvidence-based Nursing Sudtirol, Alta Adiga, ItalyNorwegian Knowledge Centre for the Health ServicesUniversity of Pennsylvania Health System Center for EB Practice Journal of Infection in Developing Countries - InternationalAgencia sanitaria regionale, Bologna, ItaliaThe German Agency for Quality in Medicine
E.g. Topical NSAIDs vs. placebo
SUMMARY OF FINDINGS TABLE -SOF TABLE
Oral NSAIDs vs. Placebo
SUMMARY OF FINDINGS TABLE-SOF TABLE
Cox-2 vs. Oral NSAIDsPatient or population: patients with OA and RA Intervention: Celecoxib Comparison: non-selective NSAIDs
Outcomes Illustrative comparative risks (95% CI)
Absolute difference
Relative effect (95% CI)
No of Participants (studies)
Quality of the evidence (GRADE)
NNT
Control group rate
Intervention rate
non-selective NSAIDs
Celecoxib
Benefits
Pain VAS. Scale from: 0 to 10 cm.
See NSAID vs placebo
See Cox vs placebo
Mean difference -0.42 cm on 0-10
cm scale (-2.4 to 1.6)
(14) O MODERATE
Not statistically significant
Harms
POBs 10%
4% (3% to 5%)
-6%
0.39 (0.31 to 0.50)
73449 (8)
HIGH
17 (15 to 20)
PUBs 10%
4% (3% to 5%)
-6%
0.41 (0.33 to 0.52)
85,955 (13)
HIGH
17 (15 to 21)
Myocardial infarction** 0.4%
0.3% (0.2% to
0.5%)
-0.1%
RR 0.80 (0.53 to 1.20)
29568 (13)
low1,2
Not statistically significant
1 RR CI ranged from 0.53 to 1.20 2 Studies were industry-funded ** Chen LC, Ashcroft DM. Risk of myocardial infarction associated with selective COX-2 inhibitors: meta-analysis of randomised controlled trials.
Pharmacoepidemiology and drug safety 2007;16:762-72.
SUMMARY OF FINDINGS TABLE-SOF TABLE
GRADE quality assessment (Grades of Recommendation, Assessment, Development and Evaluation)
EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’ FIGURE
GUIDELINES GRADE AMSTAR
©
How do you critically appraise a systematic review?
Assessing the Methodological Quality of Systematic Reviews
AMSTAR(A MeaSurement Tool to Assess
Systematic Reviews)
Acknowledgements
► BEVERLEY SHEA► Jeremy Grimshaw ► Lex Bouter► Maarten Boers► David Moher► Andy Oxman► Gord Guyatt► Peter Tugwell► Cochrane Musculoskeletal and HIV groups► CADTH► Internal and external validation teams
Main Considerations
►Why is the quality of SRs important?►How best to measure quality?►The development of AMSTAR►Future development of AMSTAR
The Perspective
► Decision makers
– Policy makers– Guidelines developers– Health professionals– Patients– Pharma
► Teachers/academics
Central Concepts
►In the context of a SR, validity is the extent to which its design and conduct minimize bias
►Methodological quality of SRs– how well the SR was conducted (AMSTAR)
►Reporting quality of SRs– how well systematic reviewers have reported their methodology
and findings (PRISMA)
AMSTAR and PRISMA:Overview
AMSTAR PRISMA
• Assesses methodological quality of systematic reviews (SRs)
• Assesses the reporting quality of systematic reviews (SRs)
• 11-item checklist • 27-item checklist
• Designed to improve the quality and conduct of SR methods
• Designed to improve transparency and consistency of reporting SR methods
• E.g.: Provides criteria for assessing a literature search as adequate or inadequate
• E.g.: Requires details of the literature search terms and sources used, but does not distinguish between adequate and inadequate searches
AMSTAR (1)
1. Was an ‘a priori’ design provided, outlining the reviewmethods prior to starting the review?The research question and inclusion criteria should be established before the
conduct of the review.
□ Yes □ No □ Can’t answer □ N/A
2. Was there duplicate study selection and data extraction? There should be at least two independent data extractors and a consensus procedure for disagreements should be in place.
□ Yes □ No □ Can’t answer □ N/A
3. Was a comprehensive literature search performed using at least two electronic sources and reasonably supplemented by other sources?At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.
□ Yes □ No □ Can’t answer □ N/A
AMSTAR (2)
4. Were the study inclusion criteria comprehensive, including
publications of all types and languages?The authors should state that they searched for reports regardless of their
publication type. The authorsshould state whether or not they excluded any reports (from the systematic review),
based on theirpublication status, language etc.
□ Yes □ No □ Can’t answer □ N/A
5. Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided.
□ Yes □ No □ Can’t answer □ N/A
6. Were the characteristics of the included studies provided?In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported.
□ Yes □ No □ Can’t answer □ N/A
7. Was the scientific quality of the included studies assessed and documented, using an established scale?‘A priori’ methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.
□ Yes □ No □ Can’t answer □ N/A
AMSTAR (3)8. Was the scientific quality of the included studies used
appropriately in formulating conclusions?The results of the methodological rigor and scientific quality should be considered in
the analysis and theconclusions of the review, and explicitly stated in formulating recommendations.
□ Yes □ No □ Can’t answer □ N/A
9. Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test for homogeneity, I²). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e. is it sensible to combine?).
□ Yes □ No □ Can’t answer □ N/A
10. Was the likelihood of publication bias assessed?An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test).
□ Yes □ No □ Can’t answer □ N/A
11. Was the conflict of interest stated?Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.
□ Yes □ No □ Can’t answer □ N/A
Feasibility
► Informal feedback on feasibilityEasy to use, understandable and easy to
apply, good teaching tool, not too difficult to score - like the individual components, some degree of difficulty with answering the question on inclusion of grey literature
► AMSTAR takes approximately 5 minutes less than the Oxman and Guyatt AMSTAR 14.9 (range 5-29) minutesOxman and Guyatt 20.3 (range 9-36)
minutesSacks 34.43 (range 22-52) minutes
Reasons to use AMSTAR?
►Value in using a common instrument►Builds on previous methodologically sound work►Good content and construct validity►Good reliability►Excellent feasibility►Published in peer reviewed journals►Has a memorable name!
Impact of AMSTAR
► Citations: 87 to date (Google Scholar)► Downloads > 6000
– http://www.plosone.org/article/info:doi%2F10.1371%2Fjournal.pone.0001350
– http://www.biomedcentral.com/content/pdf/1471-2288-7-10.pdf
► Accepted by a Canadian Association Drugs and Technology (CADTH) as the instrument to use in all assessments of methodological quality in SRs
► Published materials in support of using AMSTAR ‘Standard criteria, such as….(AMSTAR), should be used to critically appraise…’ (Andrew D Oxman*Health Research Policy and Systems 2006, 4:20)
► Modified CONSORT (Biomed Central in press) has a comment that it measures quality of reporting not methods
SUMMARY :EBM Key Concepts
6 A’s1. ASK2. ACCESS -3. APPRAISE4. AGGREGATE5. APPLY6. AUDIT
‘PICOT’ FIGURE
GUIDELINES GRADE AMSTAR
©
THANK –YOU !
QUESTIONS ?
Please complete your
GREEN EVALUATION SHEETS