47
Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit [email protected] www.rcts.org

Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit [email protected]

Embed Size (px)

Citation preview

Page 1: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Sources of Bias in Randomised Controlled Trials

David Torgerson

Director, York Trials Unit

[email protected]

www.rcts.org

Page 2: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Selection Bias - A reminder

• Selection bias is one of the main threats to the internal validity of an experiment.

• Selection bias occurs when participants are SELECTED for an intervention on the basis of a variable that is associated with outcome.

• Randomisation or other similar methods abolishes selection bias.

Page 3: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

After Randomisation

• Once we have randomised participants we eliminate selection bias but the validity of the experiment can be threatened by other forms of bias, which we must guard against.

Page 4: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Forms of Bias

• Subversion Bias

• Technical Bias

• Attrition Bias

• Consent Bias

• Ascertainment Bias

• Dilution Bias

• Recruitment Bias

Page 5: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Bias (cont)

• Resentful demoralisation

• Delay Bias

• Chance Bias

• Hawthorne effect

• Analytical Bias.

Page 6: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Subversion Bias

• Subversion Bias occurs when a researcher or clinician manipulates participant recruitment such that groups formed at baseline are NOT equivalent.

• Anecdotal, or qualitative evidence (I.e gossip), suggest that this is a widespread phenomenon.

• Statistically this has been demonstrated as having occurred widely.

Page 7: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Subversion - qualitative evidence

• Schulz has described, anecdotally, a number of incidents of researchers subverting allocation by looking at sealed envelopes through x-ray lights.

• Researchers have confessed to breaking open filing cabinets to obtain the randomisation code.

Schulz JAMA 1995;274:1456.

Page 8: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Quantitative Evidence

• Trials with adequate concealed allocation show different effect sizes, which would not happen if allocation wasn’t being subverted.

• Trials using simple randomisation are too equivalent for it to have occurred by chance.

Page 9: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Poor concealment

• Schulz et al. Examined 250 RCTs and classified them into having adequate concealment (where subversion was difficult), unclear, or inadequate where subversion was able to take place.

• They found that badly concealed allocation led to increased effect sizes – showing CHEATING by researchers.

Page 10: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Comparison of concealment

Allocation Concealment

Effect Size OR

Adequate 1.0

Unclear 0.67 P < 0.01

Inadequate 0.59

Schulz et al. JAMA 1995;273:408.

Page 11: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Case Study

• Subversion is rarely reported for individual studies.

• One study where it has been reported was for a large, multicentred surgical trial.

• Participants were being randomised to 5+ centres using sealed envelopes.

Page 12: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Case-study (cont)

• After several hundred participants had been allocated the study statistician noticed that there was an imbalance in age.

• This age imbalance was occurring in 3 out of the 5 centres.

• Independently 3 clinical researchers were subverting the allocation

Page 13: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Mean ages of groups

Clinician Experimental Control

All p < 0.01 59 63

1 p =.84 62 61

2 p = 0.60 43 52

3 p < 0.01 57 72

4 p < 0.001 33 69

5 p = 0.03 47 72

Others p = 0.99 64 59

Page 14: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Example of Subversion

05

1015202530

Recruitment Sequence

En

ve

lop

e N

um

be

r

Page 15: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Concealment

• Both the Schulz and Kjaergard considered sealed opaque envelopes to be ‘adequate’ measures of concealment.

• Envelopes can be subverted by being opened in advance.

Page 16: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

More Evidence

• Hewitt and colleagues examined the association between p values and adequate concealment in 4 major medical journals.

• Inadequate concealment largely used opaque envelopes.

• The average p value for inadequately concealed trials was 0.022 compared with 0.052 for adequate trials (test for difference p = 0.045).

Hewitt et al. BMJ;2005: March 10th.

Page 17: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

More Examples

• Berger has collected 30 case examples of potential subversion of the allocation process in clinical trials.

• Because allocation subversion is scientific misconduct it is likely that there are many other, undetected, cases.

Berger. Selection Bias and Covariate Imbalances in Randomized Clinical Trials 2005: Wiley, Chicester.

Page 18: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Recent Blocked Trial

“This was a block randomised study (four patients

to each block) with separate randomisation at each of the three centres. Blocks of four cards were produced, each containing two cards marked with "nurse" and two marked with "house officer." Each card was placed into an opaque envelope and the envelope sealed. The block was shuffled and, after shuffling, was placed in a box.”

Kinley et al., BMJ 2002 325:1323.

Page 19: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

What is wrong here?

Southampton Sheffield Doncaster

Doctor Nurse Doctor Nurse Doctor Nurse

500 511 308 319 118 118

Kinley et al., BMJ 325:1323.

Page 20: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Problem?

• If block randomisation of 4 were used then each centre should not be different by more than 2 patients in terms of group sizes.

• Two centres had a numerical disparity of 11. Either blocks of 4 were not used or the sequence was not followed.

Page 21: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Restricted allocation and subversion

• The drawback with any form of allocation restriction is that it allows some prediction.

• Simple randomisation has no ‘memory’ of the previous allocation. In contrast, blocked allocation allows the probability of an allocation to be linked to the previous allocation.

• Merely guessing that the next allocation will be the opposite of the previous one will result in a prediction more accurate than by chance.

• This can, in theory, allow subversion.

Page 22: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Possible subversion

• In a RCT of rehabilitation for the treatment of hip fracture gross baseline imbalances were detected favouring the control group.

• Secure telephone allocation had been used. But blocked allocation, size 6, had been used.

• Exploratory analysis of imbalances suggested partially successful prediction of block allocation.

Turner J. 2002, Unpublished PhD Thesis, University of York.

Page 23: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Wither restricted allocation?

• Simple randomisation followed by analysis of covariance (ANCOVA) is as efficient as restricted randomisation and ANCOVA for sample sizes > 50.

• Restricted allocation increases risk of prediction and predictability.

• For large trials simple allocation followed by ANCOVA reduces risk of prediction.

Rosenberger WF, Lachin JM. Randomisation in clinical trials: Theory and practice. Wiley Interscience, 2002, John Wiley and Sons, New York.

Page 24: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Subversion - more evidence

• In a survey of 25 researchers 4 admitted to keeping ‘a log’ of previous allocations to try and predict future allocations.

Brown et al. Stats in Medicine, 2005,24:3715.

Page 25: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Testing for subversion

• Comparison of baseline characteristics may help if subversion is suspected. Although this will only identify gross subversion.

• If blocked allocation is used a statistical test – Bergner-Exner test, may help identify subversion.

Page 26: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Concealment: Recommendations

• Allocation sequence must be independently generated and kept secret from the people who are enrolling participants.

• A secure method of giving allocation to the recruiters must be developed, opaque envelopes are inadequate.

Page 27: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Subversion - summary

• Appears to be widespread.

• Secure allocation usually prevents this form of bias.

• Need not be too expensive.

• Essential to prevent cheating.

Page 28: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Secure allocation

• Can be achieved using telephone allocation from a dedicated unit.

• Can be achieved using independent person to undertake allocation.

Page 29: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Technical Bias

• This occurs when the allocation system breaks down often due a computer fault.

• A great example is the COMET I trial (COMET II was done because COMET 1 suffered bias).

Page 30: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

COMET 1

• A trial of two types of epidural anaesthetics for women in labour.

• The trial was using MIMINISATION via a computer programme.

• The groups were minimised on age of mother and her ethnicity.

• Programme had a fault.

COMET Lancet 2001;358:19.

Page 31: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

COMET 1 – Technical Bias

AGE Traditional Combined Low dose

Total 388 335 331

<25 years 13 (3%)

179 (53%)

173 (52%)

Page 32: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

COMET II

• This new study had to be undertaken and another 1000 women recruited and randomised.

• LESSON – Always check the balance of your groups as you go along if computer allocation is being used.

Page 33: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Attrition Bias

• Usually most trials lose participants after randomisation. This can cause bias, particularly if attrition differs between groups.

• If a treatment has side-effects this may make drop outs higher among the less well participants, which can make a treatment appear to be effective when it is not.

Page 34: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Attrition Bias

• We can avoid some of the problems with attrition bias by using Intention to Treat Analysis, where we keep as many of the patients in the study as possible even if they are no long ‘on treatment’.

Page 35: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Selection bias after randomisation

• Selection bias is avoided if ALL participants who are randomised are completely followed up.

• Often there is some attrition – after randomisation some refuse to continue to take part.

• Or some may refuse the intervention but can still be tracked – IMPORTANT to distinguish between these.

Page 36: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

What is wrong here?

Random allocation

160 children 8 from

Each school

76 children allocated to control 76 allocated to intervention group

1 school 8 children withdrew

N = 17 children replaced following

discussion with teacher

Page 37: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Ascertainment Bias

• This occurs when the person reporting the outcome can be biased.

• A particular problem when outcomes are not ‘objective’ and there is uncertainty as to whether an event has occurred.

• Example, of homeopathy study of histamine, showed an effect when researchers were not blind to the allocation but no effect when they were.

• Multiple sclerosis treatment appeared to be effective when clinicians unblinded but ineffective when blinded.

Page 38: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Resentful Demoralisation

• This can occur when participants are randomised to treatment they do not want.

• This may lead to them reporting outcomes badly in ‘revenge’.

• This can lead to bias.

Page 39: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Resentful Demoralisation

• One solution is to use a patient preference design where only participants who are ‘indifferent’ to the treatment they receive are allocated.

• This should remove its effects.

Page 40: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Hawthorne Effect

• This is an effect that occurs by being part of the study rather than the treatment. Interventions that require more TLC than controls could show an effect due to the TLC than the drug or surgical procedure.

• Placebos largely eliminate this or TLC should be given to controls as well.

Page 41: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Analytical Bias

• Once a trial has been completed and data gathered in it is still possible to arrive at the wrong conclusions by analysing the data incorrectly.

• Most IMPORTANT is ITT.

• Also inappropriate sub-group analyses is a common practice.

Page 42: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Intention To Treat

• Main analysis of data must be by groups as randomised. Per protocol or active treatment analysis can lead to a biased result.

• Those patients not taking the full treatment are usually quite different to those that are and restricting the analysis can lead to bias.

Page 43: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Sub-Group Analyses

• Once the main analysis has been completed it is tempting to look to see if the effect differs by group.» Is treatment more or less effective in women?» Is it better or worse among older people?» Is treatment better among people at greater

risk?

Page 44: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Sub-Groups

• All of these are legitimate questions. The problem is the more subgroups one looks at the greater is the chance of finding a spurious effect.

• Sample size estimations and statistical tests are based on 1 comparison only.

Page 45: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Sub-Group and example.

• In a large RCT of asprin for myocardial infarction a sub-group analysis showed that people with the star signs Gemini and Libra aspirin was INEFFECTIVE.

• This is complete NONSENSE!

• This shows dangers of subgroup analyses.

Lancet 1988;ii:349-60.

Page 46: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Sub groups

• To avoid spurious findings these should be pre-specified and based on a reasonable hypothesis.

• Pre-specification is important avoid data dredging as if you torture the data enough it will confess.

Page 47: Sources of Bias in Randomised Controlled Trials David Torgerson Director, York Trials Unit djt6@york.ac.uk

Summary

• Despite the RCT being the BEST research method unless expertly used it can lead to biased results.

• Care must be taken to avoid as many biases as possible.