Do we understand our data? Evaluating comprehension and

usefulness of statistical methods for continuous outcomes in meta-analyses

Furqaan Sadiq1, Reem A. Mustafa1,2, Bradley C. Johnston2, Gordon H. Guyatt2

1 UMKC School of Medicine, 2McMaster University, Canada

OBJECTIVE

INTRODUCTION

• Clinicians rely on summary estimates from systematic reviews (SR) and meta-analyses for clinical decision making.

• Interpreting a treatment’s effects – large or small – can be difficult. • When pooling results of trials, authors of SR report the differences between

intervention and control groups in standard deviation units of Standardized Mean Difference (SMD).

• Presenting results as SMD is the longest standing and most widely used approach.• SMD values 0.20 represents a small difference, 0.50 represents a moderate

difference and 0.80 represents a large difference.

• To determine clinicians’ understanding and perceptions of 6 different approaches to the presentation of continuous outcome data in meta-analyses.

• The six include SMD, Minimal Important Difference Units, Natural Units, Relative Risk, Risk Difference and Ratio of Means

METHODS

DISCUSSION• UMKC clinicians best understood continuous outcomes when presented as

dichotomies (relative and absolute risk differences) and also found these presentations most useful.

• Presenting results as SMD, the longest standing and most widely used approach, was poorly understood and not perceived as useful.

Strengths: • This is one of the first studies to survey physicians about their understanding and

perceived usefulness of different continuous outcome presentations.• We used both perceived usefulness and percent of correct answer • We surveyed both internal medicine and family medicine physicians• We achieved an excellent response rate from the physicians surveyed

Limitations: • Data is from a single center, which may represent a biased view. However, the

preliminary results from other centers support the same findings. • There may have been a learning effect. However, we did not observe any

systematic differences in responses based on forms with different orders.• Surveys were distributed in an academic institution, so the results may not be

generalized to practicing physicians outside academia.

• 63 clinicians responded (39 FM and 24 IM), all of which provided completed surveys (95.5% response rate)

• Data analysis entailed calculating proportion of participants answering correctly for small and large effects, along with number of participants who favored each approach. 95% Confidence Interval was constructed for each entity.

• Risk Difference was the approach best understood by clinicians, followed by the Ratio of Means and Relative Risk (Table 2).

• Clinicians generally found dichotomous presentation of continuous outcomes (Relative Risk; Risk Difference) very useful, and other approaches less useful (Table 3).

ACKNOWLEDGEMENTSFunding: NoneWe thank all participant clinicians who answered our surveyReferences: Cochran Review Handbook 2011

RESULTSTable 3. Perceived Usefulness of the Presentation of Continuous Outcomes, n = 63

RESULTS

Table 2. Understanding of the Presentation of Continuous Outcomes, n = 63

Survey design: Participants received paper-based, self-administered surveys presenting summary estimates of a hypothetical intervention versus placebo for chronic pain, with estimates demonstrating either a small effect or large effect for each of the 6 presentation approaches. We asked 6 questions addressing understanding and 6 questions addressing preferences. We randomized participants to size of effect and order.

Participants: As part of a larger multicenter international study, we invited 66 staff, residents, and trainees in family medicine (FM) and internal medicine (IM) academic programs at University of Missouri-Kansas City (UMKC) to participate, 41 FM resident and staff and 25 IM residents.

We evaluated 6 common presentation approaches found in systematic reviews and meta-analyses. (1) Standard deviation units – standardized mean difference (SMD)(2) Minimal important difference units (MID)(3) Conversion into natural units of the most commonly used instrument. (4) Conversion to relative effects, calculation of relative effects (e.g. relative risk) (5) Conversion to absolute effects, calculation of absolute effects (e.g. risk difference and the corresponding number needed to treat)(6) Ratio of means or ratio of change - ratio of means (RoM) or ratio of change (RoC) between the intervention and control groups

 

Approach Mean (SD) (95%CI)

SMD 2.95 (1.43) [2.60 – 3.30]

MID 2.92 (1.53) [2.55 – 3.30]

Natural units 3.75 (1.61) [3.35 - 4.15]

Relative Risk 4.16 (1.69) [3.74 - 4.58]

Risk Difference 4.30 (1.73) [3.87 – 4.72]

Ratio of Means 3.95 (1.61) [3.55 - 4.35]

Approach N (%) correct (95% CI)

SMD 16 (25.4%) (16.3% - 37.3%)

MID 9 (14.3%) [7.71% - 25.0%]

Natural units 10 (15.9%) [8.85% - 26.8%]

Relative Risk 21 (33.3%) [22.9% - 45.6%]

Risk Difference 24 (38.1%) [27.1% - 50.5%]

Ratio of Means 22 (34.9% [24.3% - 47.3%]

Table 1. Demographic characteristics of respondents , n = 63

Characteristic N (%)

Gender MaleFemale

3231

(50.8%)(49.2%)

Specialty Internal MedicineFamily Medicine

2439

(38.1%)(61.9%)

Professional status StaffTrainee

657

(9.6%)(90.4%)

Year graduated from medical school

Before 19901990-19992000-20092010and after

31

1247

(4.8%)(1.6%)

(19.0%)(74.6%)

Training in Health Research Methods or Epidemiology

Never completed a formal courseCompleted a formal course but no degreeHave master/PhD degree in HRM

47160

(74.6%)(25.4%)

(0%)

1-7 scale with higher numbers indicating higher preference