Weighing the Evidence:Critical Appraisal Fundamentals

Lita Anglin, MSISCenter for Innovations in the Advancement of Care

Joey Nicholson, MLIS, MPH, CPHNYU Health Sciences Library

March 12, 2019

Learning Objectives & COI Disclosure

In compliance with ACCME Standards for Commercial Support of CME activities, we as the speakers have no relevant financial relationships to disclose.

• What is critical appraisal?• How is it situated within the EBP process?• Common question types/study methodology• Practicing the framework

• Implicit, rapid, full• Critical appraisal tools

What is critical appraisal?

What is critical appraisal? Why would I use it?

A structured, systematic approach to evaluating information for validity, scientific rigor or quality, and usefulness to practice. The process helps clinicians to make clinical decisions based on evidence.

“Using critical appraisal skills and tools enables users of research evidence to reach their own judgements.”

CASP, Critical Appraisal Skills Program

Reading to understand

Reading critically is an active way of reading – a deeper, more complex engagement with a text. It is analyzing, interpreting, and evaluating. It is looking at the text to see the patterns of argument, the validity of the argument, and the value of the work.

Jennifer Duncan. Reading Critically. The Writing Centre, University of Toronto Scarborough https://utsc.utoronto.ca/twc/

Asking questions

• Divergent (background)

• Convergent (foreground/clinical)

Three levels of critical appraisal

• Implicit critical appraisal• Rapid critical appraisal• Full critical appraisal

Try this!

Part of the EBP Framework

Evidence-based practice: step by step. American Journal of Nursing’s 12-part series from the Arizona State University College of Nursing and Health Innovation's Center for the Advancement of Evidence-Based Practice.

Melnyk BM, Fineout-Overholt E, Stillwell SB, Williamson KM

November 2009January 2010March 2010May 2010July 2010 – Critical Appraisal of the Evidence, Part ISeptember 2010 – Part IINovember 2010 – Part IIIJanuary 2011March 2011May 2011July 2011September 2011

One Part of the EBP Framework0. Cultivate a spirit of inquiry.

1. Ask clinical questions in PICOT format.

2. Search for the best evidence.

3. Critically appraise the evidence.

4. Integrate the evidence with clinical expertise and patient preferences and values.

5. Evaluate the outcomes of the practice decisions/changes based on evidence.

6. Disseminate EBP results.American Journal of Nursing, Evidence-based Practice Step by Step series.

13

4 explanations for a study finding

13

Truth

Bias

Chance

Confounding

3 overarching questions to ask to get to the truth1. Are the results of the study valid and reliable?

i.e. Do they describe appropriate study methodology and show how they are accounting for bias, confounding, and chance? Are they measuring what they set out to measure?2. What are the results and how significant are they?

i.e. Do they report the size of effect of their outcome (odds ratio, relative risk, mean difference) and do they state how precise it is (confidence interval)?3. Do they apply to my situation/patient?

i.e. Is the population similar? Is the setting similar? Could similar results be expected in this setting?

What is Evidence?

The Metropolitan Museum of Art, New York. The Great Pyramid and The Great Sphinx, Francis Frith, 1858.

Building the pyramid of evidence

Level VII: Expert opinion or consensus

Level VI: Qualitative or descriptive studies (including evidence implementation projects)

Level V: Systematic review of qualitative or descriptive studies

Level IV: Case-control or cohort studies

Level III: Controlled trials, not randomized

Level II: Randomized controlled trials

Level I: Systematic reviews or meta-analysis

Many Levels of Evidence

What makes the study designs different?

From the Centre for Evidence-Based Medicine: https://www.cebm.net/2014/04/study-designs/

Systematic Reviews/Meta-Analyses – Level 1Best for: Synthesizing evidence for effects of an intervention across multiple studies – All Kinds of Questions

Advantages:• should include critical appraisal of studies examined;• pulls evidence into one place for easier decision making;• can help account for biases, confounding, and chance across lower level study designs.

Disadvantages:• highly susceptible to publication bias; • inconsistent reporting standards for results;• inconsistent adherence to strict methodology.

Systematic Reviews/Meta-Analyses – Level 1

Controlled Trials – Levels 2 and 3Best for: Studying the effect of an intervention on an outcome – Questions of Therapy

Advantages:• unbiased distribution of confounders;• blinding more likely;• randomization facilitates statistical analysis.

Disadvantages:• expensive: time and money;• volunteer bias;• ethically problematic at times.

Controlled Trials – Levels 2 and 3

Cohort Study – Level 4Best for: studying risk factors for developing a given outcome – Questions of Prognosis, Etiology (Cause), Therapy

Advantages:• ethically safe;• subjects can be matched;• can establish timing and directionality of events;• eligibility criteria and outcome assessments can be standardized;• administratively easier and cheaper than RCT.

Disadvantages:• controls may be difficult to identify;• exposure may be linked to a hidden confounder;• blinding is difficult;• randomization not present;• for rare disease, large sample sizes or long follow-up necessary.

Cohort Study – Level 4

Case-Control Study – Level 4Best for: studying rare disorders or circumstances – Questions of Prognosis, Etiology (Cause), Therapy

Advantages:• quick and cheap;• only feasible method for very rare disorders or those with long lag between exposure and outcome;• fewer subjects needed than cross-sectional studies.

Disadvantages:• reliance on recall or records to determine exposure status;• confounders;• selection of control groups is difficult;• potential bias: recall, selection.

Case-Control Study – Level 4

Qualitative and Meta-Syntheses – Levels 5 and 6Best for: Describing a population or problem and highlighting the variables around it to quantify the scope of the problem or explore behaviors/feelings about the problem

Advantages:• Cheap and simple;• Ethically safer.

Disadvantages:• Establishes association at most, not causality;• Highly susceptible to recall bias;• Confounders may be unaccounted for;• May not be representative.

Qualitative and Meta-Syntheses – Levels 5 and 6

The type of question drives study designClinical Question Type Most Likely Study Design You’ll Find

All Questions Systematic Reviews/Meta-Analyses > RCTs

Therapy Randomized Controlled Trials (RCTs)

Etiology (Cause) Cohort Studies

Diagnosis Prospective comparison to gold standard

Prevention Cohort Studies > Case-Control

Prognosis Cohort Studies > Case-Control > Case Series

Meaning Qualitative Studies

Quality Improvement RCTs

Cost Economic Evaluation Studies

Are the results of the study valid and reliable?i.e. Do they describe appropriate study methodology and show how they are accounting for bias, confounding, and chance? Are they measuring what they set out to measure?

What are the results and how significant are they?i.e. Do they report the size of effect of their outcome (odds ratio, relative risk, mean difference) and do they state how precise it is (confidence interval)?

Do they apply to my situation/patient?i.e. Is the population similar? Is the setting similar? Could similar results be expected in this setting?

Practicing critical appraisal

Implicit critical appraisal - quality signals• Quality of the publication• Currency• Scope of the journal• Funding source• Study design• Sample size• Who is citing the work

Let’s try this!

Rapid screening, CASP approachArticle Is the study valid? What are the results? Are the results useful?

Jones 2019

Smith, Thomas 2016

Geller and Young, 2017

Harrison et al 2019

Rapid critical appraisal, Melnyk et al.

Evidence-Based Practice Step by Step: Critical Appraisal of the Evidence: Part I Fineout-Overholt E, Melnyk B; Stillwell,S; Williamson, K. AJN, American Journal of Nursing. 110(7):47-52, July 2010.

Article Level of Evidence How Well Conducted Usefulness to Practice

Jones 2019

Smith, Thomas 2016

Geller and Young, 2017

Harrison et al 2019

Rapid critical appraisal of systematic reviews

Evidence-Based Practice, Step by Step: Critical Appraisal of the Evidence: Part II: Digging deeper—examining the "keeper" studies.

Fineout-Overholt, Ellen; Melnyk, Bernadette Mazurek; Stillwell, Susan B.; Williamson, Kathleen M.

AJN The American Journal of Nursing110(9):41-48, September 2010.

doi: 10.1097/01.NAJ.0000388264.49427.f9

Critical appraisal, full

Critical appraisal tools

How can appraisal tools help?

Critical appraisal tools guide reviewers and improve the objectivity of the review process. They help us remember the task at hand, the criteria we are evaluating, and help us focus on very specific components and methods of the evidence. They ensure that the review process is (mostly) uniform among reviewers and that the process is documented and reproducible. Appraisal tools can help us approach evidence with less bias.

What are some common tools and when are they used?

Checklists

CEBM, The Centre for Evidence-Based Medicine (Oxford)

CASP, Critical Appraisal Skills Programme (Oxford) (8 different lists)

JBI Critical Appraisal Tools (University of Adelaide/Joanna Briggs Institute)

BMJ Best Practice – Critical Appraisal Checklists

Tools for reviewers

Covidence, JBI SUMARI, Excel

JBI

Thank you!