1

Foolish & Fatal Flaws

When medical science goes bad

2

Outline

Background on statistical and methodological error On Pigs and PCCs

Crossing the species barrier Correct group assignment

What? You mean some of the control patients actually got transfused?

Large starting group consents, but few randomized & deceptive statistics You mean only 6.8% of patients consenting to the trial

were actually randomized? You mean there are missing statistical calculations?

‘Temporal ambiguity’ Outcome before exposure – What? The patient got

pneumonia before the transfusion?

3

The use of statistics in medical diagnoses and biomedical research may affect whether individuals live or die, whether their health is protected or jeopardized, and whether medical science advances or gets sidetracked. [...] Because society depends on sound statistical practice, all practitioners of statistics, whatever their training and occupation, have social obligations to perform their work in a professional, competent, and ethical manner.”[Ethical Guidelines for Statistical Practise,

American Statistical Association,1999].

4

When transfusion medicine gets sidetracked Recombinant factor VIIa Transfusion-related

immunomodulation Formula-driven resuscitation Albumin use in critical care RBC transfusion in patients with

ischemic heart disease

5

Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49

Study design Failure to a priori define outcomes Failure to perform sample calculations Failure to blind (or disclose who the blinding was done) Control and treatment groups not comparable

Data analysis Messing up on simple statistical tests – i.e. using a t-test

without meeting test requirements or appropriate correction tests

When comparing multiple groups – can’t use ‘two-group’ tests

Post-hoc subgroup analysis – ‘shopping’ for statistically significant results

6

Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49

Documentation ‘Where appropriate’ use of a statistical test Failure to state the number of ‘tails’ used

for a specific analysis Presentation

Correct use of standard deviation vs. 95% confidence interval vs. inter-quartile ranges

Median vs. mean P values – record exactly – not ‘ns’, ‘<0.05’,

‘>0.05’ – no cheating when you round your p values

7

Incongruence between test statistics and P values in medical papersBMC Med Res Methodol. 2004; 4: 13.

11.6% and 11.1% of the statistical results published in Nature and BMJ respectively during 2001 were incongruent, mostly due to rounding, transcription, or type-setting errors

At least one such error appeared in 38% and 25% of the papers of Nature and BMJ, respectively

In 12% of the cases, the significance level might change one or more orders of magnitude

The frequencies of the last digit of statistics deviated from the uniform distribution and suggested ‘digit preference’ in rounding and reporting (a.k.a. lying)

8

Should see uniform distribution

9

More haste, less science?Nature. 1999 Aug 5;400(6744):498

10

Name Example

Centripetal Bias Healthcare access bias

Mode for mean bias Frequency-quantity Q to assess EtOH intake, subject reports modal rather than mean intake (closer to zero)

Obsequiousness bias Subjects alter responses in the direction they perceive is desired by the investigator

Details 74 types of medical bias

11

What are the Contributing Factors to Misuse?

Pressures to publish, produce results, or obtain grants

Career ambitions or aspirations Conflicts of interest and economic

motives Inadequate supervision, education, or

training

Gardenier JS, Resnik DB. The misuse of statistics: concepts,tools, and a research agenda. Account Res. 2002;9:65–74.

12

Bottom line: Be on the look out for medical science flaws!

13

On Pigs and PCCs

Prothrombin complex concentrate vs fresh frozen plasma for reversal of dilutional coagulopathy in a porcine trauma model.

Dickneite G, Pragst I.Brit J Anesthesia 2009; 102: 345-

54. The study was funded and conducted by representatives of the company that makes the PCC product

14

The study design

47 pigs, 20-30 kg, anesthetized, 70% isovolemic blood loss with replacement with RBCs/HES

The pigs were randomized to (5-7 each group): 15 mL/kg of porcine FFP 40 ml/kg of porcine FFP 25 U/kg of human-derived PCC (Beriplex P/N) 15 mL/kg saline

Porcine FFP was used instead of human FFP because infusion of human FFP into pigs can result in transfusion reactions

Following this resuscitation, the pigs underwent a controlled injury 3 mm hole into the femur or 7 cm x 1cm incision into

the spleen

15

Outcomes – measured by blinded observers

Time to hemostasis and blood loss were monitored for 120 min after injury

Skin bleeding time (SBT) in duplicate Blood samples were collected at baseline,

after the completion of hemodilution and 5 min after study treatment administration

Coagulation factors (2/7/9/10) were measured They did not measure the non-vitamin K

dependent factors

16

The effect on the PT – all 3 work

17

Factor X – only PCC works

Human equiv11 units FFP

Expectedresults

18

Proportion with hemostasisBetter off with saline than FFP?

PCC

Saline

FFP - low

FFP – high

19

Spleen blood loss

20

Bone blood loss

Saline

FFP-low FFP-highPCC

21

Skin bleeding time

22

Fibrinogen level

2 ½ dosesOf FFP?

23

The investigators conclusion In view of the unmet clinical

need for more efficacious haemostatic agents in such patients, clinical studies are now justified to confirm the observed favorable effects of PCC in the present preclinical model system

24

A lot of unanswered questions The laboratory tests used to measure clotting factor

levels are based on human factor-deficient plasma Do the lab assays work for human and pig samples for all

factor assays performed? Perhaps this explains why PT corrects but not factor

assays Why would massive FFP exposure increase blood

loss in these pigs? Was there something wrong with their pig FFP? Is hemodilution with FFP a bad thing? Did they fail to give enough RBCs?

Why would human PCC immediately stop bleeding in these pigs and FFP would make you bleed more than saline?

25

Comment

Pig vs. Human comparison Infusion of human factors into pig recipients via PCC may

have induced enhanced coagulation compared with what those same proteins would achieve in a human recipient?

The clinical bleeding outcomes observed between animals infused with pig FFP versus animals infused with human PCC in this experiment are difficult to compare directly

The real value of PCCs compared with FFP in human trauma cannot be answered by this experimental design

If you are a pig and you get injured, you want human PCC!

26

Caution

Make sure your trauma surgeons do not take this study at face value

27

Correct group assignmentWhen you untransfused control group might be transfused

Intraoperative transfusion of 1 U to 2 U packed red blood cells is associated with increased 30-day mortality, surgical-site infection, pneumonia and sepsis in general surgery patients.

Bernard AC,Davenport DL,Chang PK, et al. J Am Coll

Surg 2009; 208: 931-37

28

Design

Prospective study of patients undergoing major surgical procedures at 121 hospitals

Nurses prospectively collected preoperative, intraoperative and postoperative variables for 30 days after the operation on the first 40 operations in each 8-day cycle

Database was queried for 05-06 for all general surgery procedures

29

Transfused vs. not transfused ‘Transfusion’ =

number of PRBC units transfused intraoperatively

Transfused >4 U RBCs is the first 72 hours post-op

2nd group = yes/no definition, not number of units

30

Time course

Intraoperative 30 days

72 hours

>4 unitsyes/no

>4 unitsyes/no

numbertransfused

Data variable only

=Transfused

OR

31

Outcomes over 30 days

Composite surgical-site infection (superficial, deep, or organ/space)

Urinary tract infection Pneumonia (without preoperative pneumonia) Sepsis/septic shock Composite morbidity – all of the above 1 or more of 20 adverse events uniformly

defined by the ACS-NSQIP, excluding bleeding requiring transfusion

Mortality

32

Analysis

Estimated probability for a patient to receive a transfusion (propensity) was calculated by MLR analysis of all the available patient and operative risk factors

Risks for outcomes by level of intraoperative transfusion were calculated using logistic regression, with adjustment for: transfusion propensity, procedure group, and complexity other ACS-NSQIP risk factors operative duration (proxies for technique), level of transfusion received intraoperatively postoperative transfusion of >4 U PRBCs

33

Results

125,223 general surgery patients at 121 hospitals were retrieved

4,788 patients (3.8%) received intraoperative RBC or >4 in the 72 hours post

Risk variables most predictive of transfusion were inpatient procedure, procedure group, ASA class, hematocrit >38!, preoperative transfusion >4 U, emergent procedure, esophageal varices, and age

34

Results

Patients receiving a single unit of intraop RBCs had higher rates of surgical-site infections, urinary tract infection, pneumonia, sepsis/shock, composite morbidity, and 30-day mortality

After adjustment for transfusion propensity, procedure group and complexity, wound class and operative duration, and all other important risk variables, transfusion significantly (p<0.05) increased the risk of mortality (OR 1.32), composite morbidity (OR 1.23), pneumonia (OR 1.24), and sepsis/shock (OR 1.29), but not surgical-site infection

35

Their conclusions – Can’t argue with these motherhood statements RBC transfusions should be used very selectively

during surgical procedures. Mild anemia should be tolerated. Blood-conservation strategies and appropriate

indicators for transfusion should be used. Additional studies must determine the

mechanisms by which transfusion of PRBCs and other blood components contribute to poor patient outcomes.

Although deleterious effects are evident and some mechanisms have been suggested, reversible causes and effective treatments have yet to be definitively determined.

36

The flaw

Their statistical model was based on patients requiring intraoperative transfusions and if a patient was given >4 transfusion within 72 hours post-operatively.

As such, a patient could have received up to 8 transfusions in the perioperative period and would have been considered as not having a transfusion!

37

Large population screened (few enrolled)Deceptive statistics, ‘underpowered’

Safety and efficacy of recombinant activated

factor VII. A randomized placebo-controlled trial in

the setting of bleeding after cardiac surgery. Gill R, Herbertson M, Vuylsteke A, et al. Circulation 2009;

120: 21-27.industry-funded

38

Design

Phase 2 dose-escalation study

Safety and possible benefits 30 sites in 13 countries Aug 2004 - Nov 2007

placebo

Cohort 1 (n=70) 1:1 randomization

40 ug/kg rVIIa

Cohort 2a (n=51)1:2 randomization

placebo 80 ug/kg rVIIa

Cohort 3 (planned, but not done)Steering/safety Comm advised against

Placebo vs. 160ug/kg

Cohort 2b (n=51)DSMB required repeat due to safety concerns

placebo 80 ug/kg rVIIa

39

Primary endpoint

The primary endpoint was the incidence of critical serious adverse events (death, myocardial infarction, stroke, and venous thromboembolic complications)

Note: Don’t get too excited – the study is stopped early before we get the answer (maybe)

40

Patient population

Inclusion criteria: Pt admitted to CVICU 30 min (excluded patients

bleeding intraop) Post-op bleeding into drains in the cardio-thoracic cavity:

200 ml/hr or 2ml/kg for 2 consecutive hours Urgent re-op not required ‘per investigator judgment’Examples of exclusion criteria: History of CVA/DVT/PE Hereditary thrombophilia VAD, ECMO, aortic arch +/or descending thoracic aorta 1st time CABG + none or only 1 antiplatelet medication

within 5 days Unacceptable thrombotic risk ‘as per site investigator’

41

Patient outcome

Randomized (n=179)

Randomized and dosed (n=172)

Randomized and not dosed (n=7)

Placebo (n=68)Death (n=4, 6%)

40 ug/kg (n=35)Death (n=4, 11%)

80 ug/kg (n=69)Death (n=6, 9%)

Study terminated prior to introduction of cohort 3“based on the data within the expanding cardiac literature in which doses of rVIIa were in the range of 60 ug/kg” ???The time to drug administration was 2.8 hrs arriving in ICU

Consented (n=2619) 6.8%

42

Overview of adverse events

SAEs Placebo (68)

40 ug/kg (35)

80 ug/kg (69)

Death 4 (6) 4 (11) 6 (9)

Cerebral infarction

0 2 (6) 2 (3)

MI 1 (2) 3 (9) 1 (1)

PE 0 0 0

Other TEs

0 1 (3) 2 (3)

P 0.25 0.43

OR 2.16 (.6-8.1)

1.61 (.5-5.3)

43

Sample size calculationPoorly written Sample size was chosen to have <20% risk of

seeing >14 (of 35) on active versus <7 (of 35) on placebo in cohort 1 What they saw: 5/68 (7.35%) placebo vs. 10/35

(28.57%) treatment – 21.22% absolute risk increase <16.7% risk of 13 (of 34) on active versus 2 (of

17) on placebo or 7 on active versus 8 on placebo in cohorts 2a, 2b and 3, all assuming no differences and 21 events in cohort 1 and 15 events in cohorts 2a, 2b, and 3. What they saw: 5/68 (7.35%) placebo vs. 11/69

(15.94%) treatment – 8.59% absolute risk increase

Data on 1st 35 controlNot provided

44

Sample size - continued

“Additionally, the sample size was chosen to give adequate power to detect a 35% reduction in the need for any allogeneic transfusions.” The power for the efficacy evaluation is based on a

comparison of (all) placebo patients with the highest dose of rFVIIa (ie, cohort 3) – never done

11% (placebo) vs. 32% (40 ug/kg) vs. 27% (80 ug/kg) – 20% absolute risk reduction

“This simple comparison between 2 groups (86 on placebo versus 34 on rFVIIa) then has 80% power assuming 80% transfusion rate on placebo and a 35% reduction to 52%” (89% placebo, 68% on 40 ug/kg, 73% on 80 ug/kg)

45

Placebo Drug

Any SAE* 5 21

No SAE 63 83

Total 68 104

* SAE = Death, Cerebral Infarction, MI, PE, or other TE

p=0.0284, Fisher

r7a was associated with a statistically higher frequency of Serious Adverse Events

46

Efficacy

placebo

40 ug/kg

80 ug/kg

Re-op for bleeding (%) 25 14

P=0.21

12

P=0.04

Volume allogeneic blood (ml)

825 640

P=0.05

500

P=0.04

Drainage from intra-thoracic cavity 15mins-4hrs post dose (ml/hr)

51 35

P=0.76

24

P=0.018

47

Re-op rates

48

Avoiding RBCs after treatment

49

Major limitations

Statistically increased serious adverse events in rVIIa groups, although they failed to do a simple Fisher Exact test

Potential efficacy only seen at 80 ug/kg dose Very select group of 6.8% of their population Limitations:

Small sample size Industry sponsored trial Authors “compensated” by company Study never completed

50

Outcome before intervention(a.k.a. Temporal ambiguity)

Transfusion and pneumonia in the trauma

intensive care unit: An examination of the

temporal relationship. Vandromme MJ, McGwin G,

Marques MB, et al. J Trauma 2009; 67: 97-101

51

Background

Retrospective studies have found an association between transfusion in ICU patients and pneumonia

Associations between transfusion and ALI, MOF, tumor recurrence and mortality

These reports have been used as evidence to support the notion of ‘transfusion-related immunomodulation’ or TRIM

There are 2 limitations with these studies1. Retrospective and therefore this association may not

represent true causation2. The timing of transfusion is often unknown and

therefore potentially some of these patients developed pneumonia before they even received the first unit of RBCs

52

Causation or Association?

53

Design

Patients admitted to the trauma ICU at the University of Alabama University Hospital between 2004-07 who had an overall length of stay of >4 days, and who had spent 1 or more days in the ICU receiving mechanical ventilation

All transfusions to these patients were also categorized as ‘young’ (<14 days) or ‘old’ (>14 days) – as if something magical happens on day 14

All RBC units were prestorage leukoreduced The incidence of pneumonia was the primary outcome of

interest, and was defined as positive culture >105 cfu/mL on bronchoalveolar lavage

The analysis of the effect of the transfusion on pneumonia was performed for all units transfused (like all the studies that came before), and then only for units that were transfused before the first episode of pneumonia

54

Patient population

1,615 patients met their study criteria Of these patients, 73% were transfused at least 1 U

of RBC The mean number of units transfused per transfused

patient was 8.94 units A total of 270 (16.7%) patients developed

pneumonia The population was as expected for a civilian trauma

cohort (age of 40 years, 74% male, 81% with a blunt injury)

The length of stay was 22 days, with a mean duration of ventilation of 12 days

55

Pneumonia analysis

The overall adjusted relative risk for pneumonia for all transfusions was 1.99 (95% CI 1.39-2.86)

When transfusions that occurred after the diagnosis of pneumonia were excluded, the adjusted relative risk for pneumonia was no longer statistically significant (1.33, 95% CI 0.98-1.80)

56

Age of blood

They also analyzed the effect of different ages of blood on the risk of pneumonia This analysis was problematic because those who

received exclusively young blood were transfused only a mean of 4.24 units, compared to 4.74 units for those receiving old blood, and 12.07 units for those receiving a combination of old and young units (different populations of patients)

The only statistically significant relative risk they could find was that the receipt of exclusively old blood, compared to no blood, increased the risk of pneumonia, albeit only slightly (RR 1.42, 95% CI 1.01-2.02)

There was no difference between patients who received exclusively young, when compared to those who received exclusively old blood

Statistically significant ‘shopping’ may be going on here

57

Author’s conclusions

The finding of an association between transfusion and pneumonia may be an erroneous one, reflecting transfusions received as a consequence of developing pneumonia during a long stay in ICU

These authors methodologically improved on previous TRIM reports by excluding transfusions that occur after pneumonia onset, and not surprisingly the effect is less prominent

58

Some tips when reading

Read from cover to cover – don’t be an abstract reader

Think really hard about how they got their results if the results are surprising or unexpected or too good to be true

Look for statistically significant ‘shopping’ Do your own calculations Check tables for errors Read the stats section and look for missing

info

59

Any questions?