Developing guidelines What’s the difference in
process if you use GRADE? Nancy Santesso, RD, MLIS, PhD
McMaster University
Department of Health Research Methods, Evidence and Impact
19 January 2017
Disclosures
• Deputy Director of Cochrane Canada
www.canada.cochrane.org
• Member of the GRADE Working Group
www.gradeworkinggroup.org
Standard guideline development process
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
Standard guideline development process – Another full session for people interested in doing a GRADE assessment
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
Materials: available on the website
• Section about questions and outcomes
• Section about pulling out the data for outcomes
• Section about how to present synthesised data for making recommendations
GRADE approach
Background information
Good practice statements
Recommendation
Evidence to support recommendation
Recommendations are ACTION statements describing what to do For women in the first trimester, we suggest medical abortion as a first option rather than surgical abortion.
For women with confirmed ectopic pregnancy in the first trimester, we suggest surgical abortion as a first option rather than medical.
For pregnant women who are Rh negative who undergo medical abortion before 49 days from last menstrual period, we recommend to not administer Rh immunoglobulin.
Recommendations are NOT…
”Treatment X leads to these complications…”
“Clinicians should use evidence based treatments…”
What are ‘summary statements’? There is no strong evidence supporting routine antibiotic prophylaxis for medical abortion. (Level II-2) Something like…We suggest to not provide antibiotic prophylaxis for medical abortion.
Why is it not a recommendation?
What should a recommendation look like?
Institute of Medicine Standards
National Guideline Clearing House
Recommendation
Summary statements
Detailed summary of evidence
Very particular about framing the question and choosing outcomes Ask questions when…
• it’s not clear what clinicians should do – guidance is needed
• there is some controversy about what to do
• there is some evidence (e.g. not a very new procedure not available in practice yet)
Should
A or B
be used in
people with X?
Materials: Brainstorming questions
1. Do prophylactic antibiotics reduce the risk of post abortion endometritis among women having first/second trimester surgical abortion?
a) What prophylactic antibiotic regimens are associated with reduced risk of post abortion endometritis among women having first/second trimester abortion?
b) What are the risks associated with prophylactic antibiotics?
Should
A or B
be used in
people with X?
What is the A? What is the B? Who are the people?
Antibiotic prophylaxis No prophylaxis Women having 1st/2nd …
What about the risks? Where do they fit in?
1. Do prophylactic antibiotics reduce the risk of post abortion endometritis among women having first/second trimester surgical abortion?
a) What prophylactic antibiotic regimens are associated with reduced risk of post abortion endometritis among women having first/second trimester abortion?
b) What are the risks associated with prophylactic antibiotics?
Should
antibiotic prophylaxis or
no prophylaxis
be used in
women in 1st/2nd trimester…?
You will decide based on all the benefits and harms of each treatment; and burden, resource use, acceptability, feasibility
In women undergoing second-trimester medical abortion, does feticide confer any benefits over not inducing fetal death?
INSTEAD
In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed? Consider benefits, harms, resources, etc.
Should A versus B be used for medical abortion in women during the first trimester
Abortion completed with intended method
Time to completed abortion
Haemorrhage requiring blood transfusion
Haemorrhage not requiring transfusion
Surgical abortion
Side effects (GI)
Pain
Hospitalisation
Satisfaction
Acceptability
Many outcomes and more!
Decide ahead of time which ones are important to
decision making
Selective use of outcomes in recommendations
• Similar to selective reporting of outcomes in trials
• Recommendations about whether to use a treatment or not, are often made on selective use of outcomes
• We should decide ahead of time what are the most important outcomes that will determine the recommendation
Does the balance of benefits and harms favour A or B?
Choose 5-7 important outcomes – look very hard for
that evidence
Use time wisely Good decision making
Medical method A
• Increase benefits
• Increase minor infections
Medical method B
• Increase benefits
• Increase minor bleeding
Advocates for B may argue that minor infections are a more critical side effect and so should not recommend medical method A
Decide ahead of time what outcomes are important to avoid bias
Establish important outcomes early Quick survey or discussion/email
Standard guideline development process
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
• Brainstorm questions • Format in “should A versus B be used in patients with X”
• Brainstorm important outcomes (include benefits and risks) • Rate the importance of outcomes to limit to 5-7 outcomes
GRADEd recommendation? Best practice statement? Or background?
Good Practice Statements: How to identify?
Questions to decide if is better as a good practice statement
Good practice statement
Is the message really necessary in regard to actual
health care practice?
Yes many clinicians are not doing this
Questions to decide if is better as a good practice statement
Good practice statement
Is the message really necessary in regard to actual
health care practice?
Yes many clinicians are not doing this
After consideration of all relevant outcomes and
potential downstream consequences, will implementing
the good practice statement result in large net positive
consequences.
Yes, serious consequences of complications can
be avoided and information can be provided with
little to no harm but great benefit
Questions to decide if is better as a good practice statement
Good practice statement
Is the message really necessary in regard to actual
health care practice?
Yes many clinicians are not doing this
After consideration of all relevant outcomes and
potential downstream consequences, will implementing
the good practice statement result in large net positive
consequences.
Yes, serious consequences of complications can
be avoided and information can be provided with
little to no harm but great benefit
Is collecting and summarizing the evidence a poor use
of a guideline panel’s limited time and energy
(opportunity cost is large)?
We would likely have to contact hospitals for data
about women who have come for urgent
care…knowledge of women and awareness,
current practices of clinicians
Questions to decide if is better as a good practice statement
Good practice statement
Is the message really necessary in regard to actual
health care practice?
Yes many clinicians are not doing this
After consideration of all relevant outcomes and
potential downstream consequences, will implementing
the good practice statement result in large net positive
consequences.
Yes, serious consequences of complications can
be avoided and information can be provided with
little to no harm but great benefit
Is collecting and summarizing the evidence a poor use
of a guideline panel’s limited time and energy
(opportunity cost is large)?
We would likely have to contact hospitals for data
about women who have come for urgent
care…knowledge of women and awareness,
current practices of clinicians
Is there a well-documented clear and explicit rationale
connecting the indirect evidence?
Yes, we can link practice, women’s knowledge,
visits to urgent care…
What components of the history and physical are indicated in a post-abortion follow-up visit?
• Should this be a recommendation?
• Should this be a good practice statement?
• Or is this simply background information about doing a history and physical?
Is the message really necessary in regard to
actual health care practice?
Standard guideline development process
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
Systematic review of the evidence – labour intensive
• Search for reviews and/or studies
• Select studies
• Extract data from studies and assess studies
• Synthesise and assess evidence
Search for reviews and/or studies – strategic process
• Start searching for previously conducted reviews (and possibly update those reviews) – watch out for non-systematic reviews!
• Conduct specific searches to limit the number of papers
• Restrict to randomised controlled trials when possible
• Restrict to large observational studies when possible
Extract data from studies and assess studies
• Start by organising data immediately by outcomes – only do the 5-7 outcomes that were ranked as important
• Decide whether a meta-analysis or statistical synthesis of the data will be done
• If not, see example in handout
Medical abortion with A versus B in first trimester
We found 8 studies, some studies measured some of our outcomes
• Study 1 measured completed abortion and found X, it also measured haemmorhage and found Y. For pain, it found Z.
• Study 2 measured completed abortion and found X.
• Study 3 found that pain was reduced, and minor haemorrhage was increased.
Study and outcome Results Risk of bias of study
Completed abortion
Study 1 50/50 with A; 43/51 with B
Study 2 73/84 with A; 70/85 with B
Study 3 …
Haemorhage (no transfusion)
Study 1 All bleeding reported: 2/50 with A; 4/51 with B
Study 2 5/84 with A; 4/85 with B
…
Medical abortion with A versus B in first trimester
We found 8 studies, some studies measured some of our outcomes
Study and outcome Results Risk of bias of study
Completed abortion
Study 1 50/50 with A; 43/51 with B
Study 2 73/84 with A; 70/85 with B
Study 3 …
Haemorhage (no transfusion)
Study 1 All bleeding reported: 2/50 with A; 4/51 with B
Study 2 5/84 with A; 4/85 with B
…
Next Extract data from studies and assess studies
Synthesise and assess evidence • See materials and table
Part One: Make a
summary
Part One: Make a
summary
Part Two: Assess the evidence -
GRADE
GRADE: What to consider when assessing evidence?
• Do I believe the results from these studies? Risk of bias
• Are the results consistent across studies? Inconsistency
• Are the results applicable to my question? Indirectness
• Is this effect precise enough? Imprecision
• Are these all of the studies? Publication bias
• Plus large effect, dose response, opposing confounding
Other systems
We know that we consider a few more criteria
What about the number of people in
the studies?
What about results that are totally
inconsistent across the studies?
This is about evidence for one outcome!
• Do I believe the results from these studies? Risk of bias
• Are the results consistent across studies? Inconsistency
• Are the results applicable to my question? Indirectness
• Is this effect precise enough? Imprecision
• Are these all of the studies? Publication bias
• Plus large effect, dose response, opposing confounding
Prophylactic antibiotics with medical abortion
Outcome Evidence
Risk of infection 5 studies, 324 pregnant women
Completion 4 studies, 285 pregnant women
Gastrointestinal side effects
8 studies, 566 pregnant women
… 3 studies, 200 pregnant women
…
Prophylactic antibiotics with medical abortion
Outcome Effects of immediate
Absolute risk difference Certainty in effect
Risk of infection RR 0.71 (0.45 to 1.12) 9 fewer infections per 1000
HIGH
Completion RR 1.14 (0.82 to 1.59)
LOW
Gastrointestinal side effects
RR 2.00 (1.44 to 2.59) 120 more GI events per 1000
MODERATE
…
2 more completions per 1000
We are very certain about 9 fewer infections, and less certain about the GI events, and uncertain about completion – suggest antibiotics
Prophylactic antibiotics with medical abortion
Outcome Effects of immediate
Absolute risk difference Certainty in effect
Risk of infection RR 0.71 (0.45 to 1.12) 9 fewer infections per 1000
LOW
Completion RR 1.14 (0.82 to 1.59)
LOW
Gastrointestinal side effects
RR 2.00 (1.44 to 2.59) 120 more GI events per 1000
HIGH
…
2 more completions per 1000
We are uncertain about 9 fewer infections, and very certain about the GI events, and uncertain about completion – suggest no antibiotics
Important to consider the effects but also certainty in those effects when making
recommendations
More about how to GRADE in another webinar
Making recommendations
In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed? Consider benefits, harms, resources, etc.
In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed?
How large are the benefits of this medication? How large are the harms? What is the balance of benefits and harms? Will it save money or cost money to use it? Is it acceptable to provide? Is it feasible to provide? Based on the answers, the guideline group decides whether to recommend it
Strength of recommendation does not rely on the quality of the
evidence
Just one part
In women undergoing second-trimester medical abortion, should feticide or not inducing fetal death be performed?
How large are the benefits of this medication? SMALL – HIGH CERTAINTY How large are the harms? TRIVIAL – HIGH CERTAINTY What is the balance of benefits and harms? PROBABLY FAVOURS FETICIDE Will it save money or cost money to use it? LARGE COSTS Is it acceptable to provide? YES Is it feasible to provide? NOT FEASIBLE IN MOST SETTINGS WE SUGGEST NOT INDUCING FETAL DEATH
We suggest… We recommend… Based on these criteria
• How large are the benefits of this medication?
• How large are the harms?
• What is the balance of benefits and harms?
• Will it save money or cost money to use it?
• Is it acceptable to provide?
• Is it feasible to provide?
Making recommendations: Balance of all factors
For Against
Benefits and Harms Certainty in evidence
Values Acceptability
Feasibility Equity Costs
For Against
Should we recommend feticide or not inducing death? Weak/conditional recommendation against…
Weak/conditional recommendation
AGAINST A
For Against
Should we recommend feticide or not inducing death? Strong recommendation against…
Strong recommendation
AGAINST A
Additional information about interpretation
Standard guideline development process
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
Standard guideline development process – Another full session for people interested in doing a GRADE assessment
Establish group Decide questions
and outcomes Gather evidence and synthesise
Assess evidence Make
recommendation
More information available on the SOGC website