Can we trust patient reported outcomes in the absence of blinding?

Can we trust patient reported outcomes in the absence of blinding?

Fiona Campbell, Mike J. Bradburn, Amitabh Deka

Canadian Cardiovascular Society Angina Score

Re v ie w: Pe rc u ta n e o u s L a s e r My o c a rd ia l Re v a s c u la ris a tio nCo mp a ris o n : 02 PMR v s c o n tro l Ou tc o me : 03 n u mb e r imp ro v in g a t le a s t 2 CCSA c la s s e s

Stu dy Tre a tme n t Co n tro l OR (ra n d o m) We ig h t OR (ra n d o m)o r s ub -c a te g o ry n /N n /N 9 5 % CI % 9 5 % CI

Oe s te rle 42/92 11/99 17.54 6.72 [3.18, 14.21] Sto ne 35/71 26/70 18.53 1.65 [0.84, 3.22] Wh itlow 22/58 29/155 18.60 2.66 [1.36, 5.17] Sa lem 14/40 6/42 13.53 3.23 [1.10, 9.53] L e on 40/98 42/102 19.91 0.99 [0.56, 1.73] Mc Na b 8/34 5/32 11.88 1.66 [0.48, 5.74]

To ta l (9 5 % CI) 393 500 100.00 2.28 [1.25, 4.15]To ta l e v e n ts : 1 6 1 (T re a tme n t), 11 9 (Co n tro l)Te s t fo r h e te ro g e ne ity : Ch i² = 1 7 .9 3 , d f = 5 (P = 0 .0 0 3 ), I² = 7 2 .1%Te s t fo r o v e ra ll e ffe c t: Z = 2 .6 9 (P = 0 .00 7 )

0 .1 0 .2 0 .5 1 2 5 10

Fa v o u rs c o n tro l Fa v o u rs tre a tme n t

Angina Score – sensitivity analysis

Re v ie w: P e rc u ta n e o u s L a s e r My o c a rd ia l Re v a s c u la ris a tio nCo mp a ris o n : 0 2 P MR v s c o n tro l Ou tc o me : 0 4 s e n s itiv ity a n a ly s is b lin d in g

S tu d y T re a tme n t Co n tro l OR (ra n d o m) We ig h t OR (ra n d o m)o r s u b -c a te g o ry n /N n /N 9 5 % CI % 9 5 % CI

S to n e 3 5 / 7 1 2 6 / 7 0 3 6 . 5 5 1 . 6 5 [ 0 . 8 4 , 3 . 2 2 ] S a le m 1 4 / 4 0 6 / 4 2 2 1 . 2 2 3 . 2 3 [ 1 . 1 0 , 9 . 5 3 ] L e o n 4 0 / 9 8 4 2 / 1 0 2 4 2 . 2 4 0 . 9 9 [ 0 . 5 6 , 1 . 7 3 ]

T o ta l (9 5 % CI) 2 0 9 2 1 4 1 0 0 . 0 0 1 . 5 3 [ 0 . 8 4 , 2 . 7 9 ]T o ta l e v e n ts : 8 9 (T re a tme n t), 7 4 (Co n tro l)T e s t fo r h e te ro g e n e ity : Ch i² = 4 .0 1 , d f = 2 (P = 0 .1 3 ), I² = 5 0 .2 %T e s t fo r o v e ra ll e ffe c t: Z = 1 .3 8 (P = 0 .1 7 )

0 .1 0 .2 0 .5 1 2 5 1 0

F a v o u rs c o n tro l F a v o u rs tre a tme n t

PROs•Landmark development – routine use introduced into the NHS in 2009

•USA Food and Drug Administration recommend their inclusion in clinical trials

Blinding/Masking•Patients/participants•Caregivers•Outcome assessors

Evidence of bias in absence of blinding – links to previous work

•Lack of double blinding exaggerated treatment effects (ratio of odds ratios 0.83 (95% CI 0.71 to 0.96)) Schulz et al 1995

•Lack of double blinding in subjective outcomes exaggerated treatment effects (ratio of odds ratios 0.75 (95% CI 0.61 to 0.93)) Woods et al 2008

What did we add?• Focus on PROs, rather than

subjective/objective• Develop a method to look at continuous

outcomes• A focus on patient blinding

• Is the effect the same across disease types?

• If bias exists are some PROs more robust than others?

• What methods are at present being used to control for bias in the absence of blinding in subjective outcomes in Cochrane reviews?

•Cochrane Systematic Reviews

•Focused on asthma, arthritis, coronary heart disease, COPD

•1 reviewer extracted data from meta-analysis our inclusion criteria

Methods:

Inclusion criteria for the SR• had to have performed a MA• PRO and an objective measure (eg.

blood test) • trials of interventions where blinding

was possible• trials that included both an unblinded

and a blinded study in the same outcome

Data extracted• details of outcome measure • details of blinding• effects of the interventions• methods of assessment of bias in the

review

Statistical methods• Outcomes tended to be continuous – presentation

concentrates on these

• For each outcome in each review calculate:• Pooled standardised mean difference (SMD),

i.e. (T - C) / • Unblinded studies SMDU

• Blinded studies SMDB

• Impact of blinding is SMDU - SMDB

Results• Preliminary!• Not what we expected!• Limited data – 11 reviews.

1. Patient assessed and reported

Chronic Respiratory Questionnaire - dyspnea (1-7)

Chronic Respiratory Questionnaire - emotion (1-7)

Chronic Respiratory Questionnaire - fatigue (1-7)

Chronic Respiratory Questionnaire - mastery (1-7)

Chronic Respiratory Questionnaire - total (1-7)

2. Patient response but clinician reported

6 Minute Walk Test - SpO2 (%)

6 Minute Walk Test - distance (metres)

6 Minute Walk Test - end-of-test borg dyspnea score(*)

Cycle Endurance - end-of-test borg dyspnea score(*)

Cycle Endurance - exercise time (minutes)

Maximal Test - PaO2 (mmHg)

Maximal Test - end-of-test borg dyspnea score(*)

Maximal Test - energy expenditure, VO2 max (L/min)

Maximal Test - power (watts)

Maximum training work level achieved (watts)

3. Objective

Cycle Endurance - SpO2 (%)

Maximal Test - SpO2 (%)

Outcome

-0.19

0.36

0.16

0.40

0.24

0.00

-0.22

0.00

0.45

0.57

-0.12

-0.10

0.00

-0.22

-0.32

-0.59

-0.26

studies

Unblinded

0.20

0.29

0.24

0.00

0.20

0.00

-0.29

-0.10

0.91

0.58

0.43

-0.15

-0.03

-0.07

0.49

-0.33

-0.28

studies

Blinded

-0.19

0.36

0.16

0.40

0.24

0.00

-0.22

0.00

0.45

0.57

-0.12

-0.10

0.00

-0.22

-0.32

-0.59

-0.26

studies

Unblinded

0.20

0.29

0.24

0.00

0.20

0.00

-0.29

-0.10

0.91

0.58

0.43

-0.15

-0.03

-0.07

0.49

-0.33

-0.28

studies

Blinded

Treatment more beneficial in blinded studies Treatment more beneficial in unblinded studies

0-2 -1 0 1 2

SMD (unblinded) minus SMD (blinded)

COPD

1. Patient assessed and reported

Health Assessment Questionnaire (HAQ)(*)

Morning stiffness duration (min)(*)

Pain (Visual Analogue Scale 10 cm)(*)

3. Objective

CRP (g/mL)(*)

ESR (mm/hr)(*)

Hemoglobin

Outcome

0.02

-0.17

0.61

-0.38

-0.25

-0.27

studies

Unblinded

-0.23

0.48

0.13

1.51

1.08

0.79

studies

Blinded

0.02

-0.17

0.61

-0.38

-0.25

-0.27

studies

Unblinded

-0.23

0.48

0.13

1.51

1.08

0.79

studies

Blinded

Treatment more beneficial in blinded studies Treatment more beneficial in unblinded studies

0-2 -1 0 1 2

SMD (unblinded) minus SMD (blinded)

Arthritis

1. Patient assessed and reported

Patient global assessment

3. Objective

CRP

Outcome

-0.23

-0.18

studies

Unblinded

0.04

0.22

studies

Blinded

-0.23

-0.18

studies

Unblinded

0.04

0.22

studies

Blinded

Treatment more beneficial in blinded studies Treatment more beneficial in unblinded studies

0-2 -1 0 1 2

SMD (unblinded) minus SMD (blinded)

Arthritis

2. Patient response but clinician reported

Fingers-to-floor test (2nd analysis) (cm)(*)(*)

Fingers-to-floor test (cm)(*)(*)

Occiput-to-wall test (2nd analysis) (cm)(*)(*)

Occiput-to-wall test (cm)(*)(*)

3. Objective

ESR (2nd analysis) (mm/hr)(*)(*)

ESR (mm/hr)(*)(*)

Outcome

-0.16

-0.16

-0.13

-0.13

0.17

0.29

studies

Unblinded

0.08

0.09

0.03

-0.04

0.42

0.37

studies

Blinded

-0.16

-0.16

-0.13

-0.13

0.17

0.29

studies

Unblinded

0.08

0.09

0.03

-0.04

0.42

0.37

studies

Blinded

Treatment more beneficial in blinded studies Treatment more beneficial in unblinded studies

0-2 -1 0 1 2

SMD (unblinded) minus SMD (blinded)

Arthritis

Conclusion

•At present we have found no evidence that in this limited number of chronic conditions (in interventions where blinding was possible) that PROs are biased in the absence of blinding.

Discussion

•Blinding can be broken – rarely assessed and reported – maybe blinded studies were not blind so that may account for lack of effect

•Is bias previously seen in subjective outcomes observer bias?

•What are the implications for interventions where blinding is not possible?

Ways forward…

Broaden the scope to include more reviews

Develop statistical methods to pool the data

Thank you.

f.campbell@sheffield.ac.uk