EBC course10 April 2003

Critical Appraisal of the Clinical Literature:

The Big Picture

Cynthia R. Long, PhDAssociate Professor

Palmer Center for Chiropractic Research

Outline

• Overall purpose of critical appraisal• Consider “levels of evidence”• Some tips on interpreting

“significant” findings• Potentially useful references

Overall Purpose

• To critically appraise a research article in order to assess the validity of the authors’ conclusions

Sections of an Article

• Introduction: Background and explanation of rationale for study.

• Methods: How was study done? Should allow study to be replicated.

• Results: Report results (data).• Discussion: Interpret results. Draw

conclusions from results.• Abstract: Article summary.

Recommendation

• Read the abstract last when familiarizing yourself with critically appraising the literature.

• Assess evidence from reading the article, not the abstract.

• Read the abstract in deciding whether or not you are interested in the topic of the article.

Critical Appraisal

• similar across types of studies for Introduction and Discussion sections

• information in Methods sections may differ

• information in Results sections may differ

Overall Questions to Ask

• Is the study design appropriate to address the research question?

• In the Discussion Section: Are the findings... – ...consistent with the research question of

the study?

Terminology: Types of Clinical Studies

• Risk• Diagnostic• Prognostic• Intervention

Terminology:Clinical Intervention Studies

• Evaluates which treatment interventions are most useful and effective for a given clinical condition

Clinical Intervention Studies

• One-group Pre/post Study • Pilot RCT• Randomized Clinical Trial (RCT)• Systematic Review of RCTs

Levels of Evidence

designed studies

clinical observations

RCT

One-group pre/post

SystematicReviews

case series, N-of-1, time series designs

case reports

Pilot RCT

Levels of Evidence

• Is the study design appropriate to address the research question?

• In the Discussion Section: Are the findings... – ...consistent with the research question of

the study?

One-group pre/post design

• Addresses: do patients improve after treatment?

• Can’t address: is treatment effective for patients?– No comparison group– Patients may improve: natural course of

condition, in study, change in lifestyle, treatment

One-group pre/post design: Example

Hawk C, Long CR, Azad A. Chiropractic Care for Women with Chronic Pelvic Pain: A Prospective Single-Group Intervention Study. JMPT 1997;20:73-79. – Results: women improved!– Conclusion: needs further study

Pilot RCT

• Addresses: is the RCT feasible?• Can’t address: is treatment

effective for patients?– Not powered (i.e. sample size not large

enough)

• Note: often only published if RCT is determined to be unfeasible

Pilot RCT: Example 1

Hawk C, Long CR, et al. Issues in planning a placebo-controlled trial of manual methods: Results of a pilot study. J of Alt Comp Med 2002;8:21-32. – Results: recruitment not feasible;

standardization of treatment protocol difficult among multiple sites

– Conclusion: put on hold

Pilot RCT: Example 2

Bronfort G, et al. Nonoperative treatments for sciatica: A pilot study for a randomized clinical trial. JMPT 2000;23:536-544. – Results: recruitment not feasible!– Conclusion: put on hold

RCT

• Addresses: is the treatment effective for patients?

• Caveats:– Must be a powered study

• sample size must be formally justified in the Methods section (based on: effect size—minimally important clinical difference; variability of outcome measure; statistical test)

RCT: Example

Hurwitz E, et al. A randomized trial of chiropractic manipulation and mobilization for patients with neck pain: Clinical outcomes from the UCLA neck-pain study. Am J Public Health 2002;92:1634-1641. – Results: no statistically or clinically significant

differences among groups– Conclusion: cervical spine mobilization is as

effective as manipulation in reducing neck pain and related disability among chiropractic patients

Overall Questions to Ask

• In the Discussion Section: Are the findings... – ...consistent with the research question of

the study?– …consistent with the results presented? – …given in the context of current evidence?

Systematic Review of RCTs

• Addresses: is the treatment effective for patients?

• Looks at all RCTs of the treatment for patients and combines based on quality of original RCTs

RCT: Example

Bronfort G, et al. Efficacy of Spinal Manipulation for Chronic Headache: A Systematic Review. JMPT 2001;24:457-466. – Conclusions: can’t make firm conclusions

(few trials of adequate methodological quality)

RCT: Example

– Conclusions:• Cervicogenic: SMT better effect than massage• Tension-type and migraine: SMT effect

comparable with commonly used first-line prophylactic prescription meds

Terminology: Descriptive Statistics

(Results)• Patient characteristics at baseline

– Examples: mean, standard deviation, median, range, percentage

– Assess group comparability on baseline characteristics

• Assess generalizability of results to target population

Terminology: Analytical Statistics (Results)• Assess statistical significance with

confidence intervals and p-values– Within and between group differences– Make inference about target population

• Must be appropriately interpreted in the context of the research question and the study design

Terminology

• P-values and confidence intervals:– Reflects measure of effect relative to

variation and sample size

Statistical Significance

Interpreting p-values: p<0.01 statistically significant difference!

0.01p0.05 statistically significant difference

0.05<p0.10 borderline statistically significant difference

p>0.10 no statistically significant difference

“Significant” Findings

• jargon term…• Need to consider BOTH statistical

and clinical significance

Clinical Significance

• i.e. clinical importance• is defined before study is

conducted• assessed with descriptive statistics

(e.g. mean improvement in outcome measure)

Possible Scenarios

• statistically and clinically significant findings

• clinically significant, but not statistically significant

• statistically significant, but not clinically significant

Critical Appraisal References

• Useful? It depends…

• BMJ series available for free on bmj.com

• JAMA articles?• Chiropractic Research Review