1

Eduardo Ortiz, M.D., M.P.H.

National Heart, Lung, and Blood InstituteNational Institutes of Health

May 10, 2011

IOM Standards for Systematic Reviews: Finding and Assessing Individual Studies

2

Acknowledgements• Colleagues at the Division for the Application of Research

Discoveries at NHLBI: Denise Simons-Morton, MD, PhD (Division Director) Glen Bennett, MS Janet de Jesus, MS, RD Karen Donato, SM, RD Rob Fulwood, PhD, MSPH Edward Donnell Ivy, MD Chi Onyewu, MD, PhD Susan Shero, MS, RN Joylene John-Sowah, MD Sid Smith, MD Zhi-Jie Zheng, MD, PhD

3

Joint National Committee on Prevention, Detection, Evaluation, &Treatment of High Blood Pressure

JNC 7: 2003 JNC 6:  1997 JNC 5:  1992 JNC 4:  1988 JNC 3:  1984 JNC 2:  1980 JNC 1:  1976

Detection, Evaluation, &Treatment of High Blood Cholesterol in Adults (ATP - Adult Treatment Panel)

ATP III Update: 2004 ATP III: 2002 ATP II: 1993 ATP I: 1988

Clinical Guidelines on the Identification, Evaluation, & Treatment of Overweight and Obesity in Adults Obesity 1: 1998

3

NHLBI Cardiovascular Clinical Guidelines

4

Guidelines for the Diagnosis and Management of Asthma (EPR-3) EPR-3: 2007 EPR-2 Update: 2002 EPR-2:  1997 EPR-1:  1991

The Diagnosis, Evaluation, and Management of Von Willebrand Disease

2006

Management of Sickle Cell Disease

2002 1995 1989 1984

4

Other NHLBI Clinical Guidelines

5

Current Guideline Efforts High Blood Pressure Cholesterol Overweight/Obesity Adult Cardiovascular Risk Reduction Crosscutting Work Groups to support our

Adult CVD Guideline efforts: Lifestyle, Risk Assessment,

ImplementationPediatric Cardiovascular Risk Reduction Sickle Cell Disease

66

Guidelines Development Process

Expert PanelSelection

External Reviewwith Revisions

as Needed

Literature Search

ScreeningData Abstraction

Study Quality Grading

Create Evidence Tables;

Grade Body of Evidence

Dissemination Implementation

Evaluation

Evidence Statementsand

RecommendationsTopic Area

Identification

Analytic Models and Critical Questions

777

Evidence-Based Systematic Review Process Step 1 – Develop analytic framework and critical

questionsStep 2 – Establish inclusion & exclusion criteria Step 3 – Screen titles, abstracts, and full text to

identify relevant studies for inclusionStep 4 – Rate the quality of each individual studyStep 5 – Abstract data for studies rated good or fairStep 6 – Create evidence tables Step 7 – Create summary tables Step 8 – Create narrative summary Step 9 – Rate the quality of the overall body of

evidence for each critical question

888

Developing the Recommendations Step 10 – Review the evidence Step 11 – Develop evidence statements, including

rating the quality of evidence for each statementStep 12 – Develop recommendations, including

grading the recommendations and making sure they are supported by the evidence

Step 13 – GLIA assessment of recommendationsStep 14 – Public comment period, with invitations for

review Step 15 – Review comments and revise

recommendationsStep 16 – Final recommendations Step 17 – Dissemination, implementation, and

evaluation

9

Developing the Critical Questions Critical questions are developed by the

expert panels working collaboratively with the methodology team and NHLBI leads

PICOTS format

Predefined inclusion and exclusion criteria

10

Screening Each study is screened by two independent

reviewers using the I/E criteria Review titles and abstracts, followed by full

text If they disagree, they discuss and try to

reach consensus If they do not achieve consensus or need

additional input, there is 3rd party adjudication

Panel members can appeal a decision Re-assessed and adjudicated by a 3rd party, but

the panel cannot override a decision made by the reviewers.

11

Lessons Learned and ChallengesIssues:

Sometimes you get the prespecified I/E criteria wrong and have to be practical and make post-hoc adjustments

Despite your best efforts, it is sometimes difficult to figure out whether a study should be included or excluded

12

Rating the Evidence Quality of each included study is rated by

two trained independent reviewers at the time of data abstraction. No satisfactory tools were available for assessing

study quality, so we developed our own Controlled intervention studies, cohort studies,

case control studies, and systematic reviews / meta-analyses

Despite your best efforts, it is sometimes difficult to determine the quality rating of an individual study

Panel members can appeal a decision Re-assessed and adjudicated by a 3rd party, but

the panel cannot override a decision made by the reviewers.

13

Lessons Learned and ChallengesWe used one abstractor and a reviewer to

check the abstracted data for accuracy and completeness Consistent with the Buscemi study, we

experienced a substantial number of errors

Dependent on the individual abstractor and reviewer

Takes a lot of time and can create a bottleneck

Using two independent abstractors would not have been feasible

14

Rating the Evidence Overall body of evidence is rated for each

critical question using a standardized rating instrument We reviewed GRADE, USPSTF, ACC-AHA, and

many other systems Hybrid model similar to USPSTF High, Moderate, Low

We need a better, standardized, user-friendly approach to rating evidence and grading the strength of recommendations

15

Lessons Learned and Challenges

For guidelines, you need to answer many questions

Conducting multiple systematic reviews to answer these questions is very challenging and requires a lot of time, effort, expertise, manpower, and money

How do we develop high-quality, credible systematic reviews to support our guidelines that can be completed and updated in a timely manner? Is there a sweet spot to aim for between evidence-based rigor and practicality?

16

Lessons Learned and Challenges

Standard 3.2 – Take action to address potentially biased reporting of research results Grey literature and other sources of unpublished studies Contacting researchers and asking them to clarify study

related information Asking study sponsors to submit unpublished data Conference abstracts Studies not published in English

It takes a lot of time and effort to search the published literature and conduct all the other steps in the SR process without searching all these additional resources

Some of these issues hopefully will be addressed by investigators and publishers, as it is not realistic to expect most groups developing SRs to be able to do all this

17

Lessons Learned and Challenges

Updating searches during and after the review process Very important, but once again practical

considerations come into play A considerable amount of time can lapse between

the initial search and completion of the review If you have to go back and update the material, it

is not just the search that has to be repeated, but all the other steps in the EB review process, which then takes more time and can lead to a vicious cycle.

How do we deal with this from a practical perspective, yet maintain a reasonable time frame and budget?

Lessons Learned and Challenges Lots of personnel, time, effort and costs to screen studies

for inclusion/exclusion, assess study quality, adjudicate decisions, abstract data, create evidence tables and summaries, etc.

If you don’t have all the needed expertise to do this in-house, contracting out the work can be challenging High costs, coordination, and decision-making across

organizations and individuals Getting screeners, reviewers, and abstractors that have

enough methodological expertise and clinical knowledge to understand important contextual issues and other nuances can be a challenge, especially when conducting multiple SRs

Variability in the quality of the reviewers and methodologists Reviewers and methodologists can differ in their

knowledge, perspectives, biases, attention to detail, etc., so the quality of reviews can vary, depending on the individuals

Final CommentsSupport the IOM recommendationsHopefully will improve the quality and

consistency of systematic reviewsReport is comprehensive and represents

an ideal process, but it would have been helpful to provide more practical or “at a minimum” recommendations to factor in real-world limitations facing most organizations

Stronger linkage between the SR and CPG reports, including more practical recommendations to assist those of us who conduct SRs for the purpose of developing guidelines

20

Thank you!

Eduardo Ortiz, M.D., M.P.H.

NHLBI Guideline Information: http://www.nhlbi.nih.gov/guidelines/index.htm