Click here to load reader

Avoiding bias in RCTs David Torgerson Director, York Trials Unit [email protected]

  • View
    213

  • Download
    1

Embed Size (px)

Text of Avoiding bias in RCTs David Torgerson Director, York Trials Unit [email protected]

  • Slide 1
  • Avoiding bias in RCTs David Torgerson Director, York Trials Unit [email protected] www.rcts.org
  • Slide 2
  • A reminder Randomised trials are needed for 4 reasons: Avoiding selection bias; Controlling for temporal changes; Controlling for regression to the mean; Basis for statistical inference.
  • Slide 3
  • Background Randomisation, when undertaken properly, prevents selection bias. Selection bias occurs when participants are allocated in such a way that allocation correlates with outcome. Selection bias is one of the main threats to validity that randomisation seeks to avoid. However, forms of selection and other sources of bias can still undermine the validity of a RCT.
  • Slide 4
  • Non-random methods: Alternation Alternation is where trial participants are alternated between treatments. EXCELLENT at forming similar groups if alternation is strictly adhered to. Problems because allocation can be predicted and lead to people withholding certain participants leading to selection bias.
  • Slide 5
  • Non-Random Methods Quasi-Alternation Dreadful method of forming groups. This is where participants are allocated to groups by month of birth or first letter of surname or some other approach. Can lead to bias in own right as well as potentially being subverted.
  • Slide 6
  • Example: Quasi alternation Before mailing, recipients were randomized by rearranging them in alphabetical order according to the first name of each person. The first 250 received one scratch ticket for a lottery conducted by the Norwegian Society for the Blind, the second 250 received two such scratch tickets, and the third 250 were promised two scratch tickets if they replied within one week.(Finsen and Storeheier, Biomed Central 2006)
  • Slide 7
  • Randomisation Some text books typically suggest the use of random number tables or coin tossing to allocate participants.
  • Slide 8
  • What is wrong with that? Coin tossing there is no audit trail. We have to take your word for what you did. Random number tables are open and again we have to take the researchers word that they did what the said. A separate computer allocation system from a third party is best.
  • Slide 9
  • Simple or restricted? Simple allocation is best when the number of units to be allocated >50. Because it is always unpredictable it is difficult to subvert or sabotage. Restricted randomisation has statistical advantages for
  • Slide 10
  • Will people subvert the allocation? Schulz [1] has described, anecdotally, a number of incidents of researchers subverting allocation by looking at sealed envelopes through x-ray lights. Researchers have confessed to breaking open filing cabinets to obtain the randomisation code. In a survey [2] of 25 researchers 4 admitted to keeping a log of previous allocations to try and predict future allocations. [1] Schulz JAMA 1995;274:1456. [2] Brown et al. Stats in Medicine, 2005,24:3715.
  • Slide 11
  • Case Study Subversion is rarely reported for individual studies. One study where it has been reported was for a large, multicentred surgical trial. Participants were being randomised to 5+ centres using opaque, sequentially numbered, sealed envelopes.
  • Slide 12
  • Case-study (cont) After several hundred participants had been allocated the study statistician noticed that there was an imbalance in age. This age imbalance was occurring in 3 out of the 5 centres. Independently 3 clinical researchers were subverting the allocation.
  • Slide 13
  • Mean ages of groups Kennedy & Grant. 1997;Controlled Clin Trials 18,3S,77-78S
  • Slide 14
  • Recent Blocked Trial This was a block randomised study (four patients to each block) with separate randomisation at each of the three centres. Blocks of four cards were produced, each containing two cards marked with "nurse" and two marked with "house officer." Each card was placed into an opaque envelope and the envelope sealed. The block was shuffled and, after shuffling, was placed in a box. Kinley et al., BMJ 2002 325:1323.
  • Slide 15
  • What is wrong here? SouthamptonSheffieldDoncaster DoctorNurseDoctorNurseDoctorNurse 500511308319118 Kinley et al., BMJ 325:1323.
  • Slide 16
  • Problem? If block randomisation of 4 were used then each centre should not be different by more than 2 patients in terms of group sizes. Two centres had a numerical disparity of 11. Either blocks of 4 were not used or the sequence was not followed.
  • Slide 17
  • Randomisation summary Very important to avoid possible problems of subversion: Who do you trust? Need independent allocation, third party, need to be convincing.
  • Slide 18
  • Selection bias after randomisation Selection bias is avoided if ALL participants who are randomised are completely followed up. Often there is some attrition after randomisation some refuse to continue to take part. Or some may refuse the intervention but can still be tracked IMPORTANT to distinguish between these.
  • Slide 19
  • What is wrong here? Random allocation 160 children 8 from Each school 76 children allocated to control 76 allocated to intervention group 1 school 8 children withdrew N = 17 children replaced following discussion with teacher
  • Slide 20
  • Selection bias POSSIBLE selection bias has been introduced in this trial. About 10% of the sample have been non- randomly allocated to the interventions. What should have happened? The authors should have retained the 17 refusers in an intention to teach (treat) analysis.
  • Slide 21
  • Ascertainment bias This occurs when the person reporting the outcome can be biased. A particular problem when outcomes are not objective and there is uncertainty as to whether an event has occurred. Example, of homeopathy study of histamine, showed an effect when researchers were not blind to the allocation but no effect when they were. Multiple sclerosis treatment appeared to be effective when clinicians unblinded but ineffective when blinded.
  • Slide 22
  • Avoiding ascertainment bias Post-tests should be administered by someone who is unaware of the group allocation. Pre-tests before allocation is made. Record searches (e.g., arrest records) should be done by a researcher blind to the participants group allocation.
  • Slide 23
  • Resentful Demoralisation This can occur when participants are randomised to treatment they do not want. This may lead to them reporting outcomes badly in revenge. This can lead to bias.
  • Slide 24
  • Resentful Demoralisation One solution is to use a participant preference design where only participants who are indifferent to the treatment they receive are allocated. This should remove its effects.
  • Slide 25
  • Example of preference interaction SPRINTER was an RCT of treatments for neckpain. Two treatments: a Brief Intervention (1-2 sessions with a physio using CBT) vs usual care (5+ sessions). BEFORE randomisation we asked patients their treatment preference.
  • Slide 26
  • SPRINTER Preferences In SPRINTER preferences were mixed 53% did not have a preference; 16% wanted brief intervention; 31% wanted usual care. ALL patients were randomised IRRESPECTIVE of their preference.
  • Slide 27
  • Patient Flow and 12 month Results - SPRINTER Overall 12 month improvement -0.840 Overall 12 month improvement -2.825
  • Slide 28
  • SPRINTER Results
  • Slide 29
  • Examples of educational trials As part of a collaboration between educational researchers and trial methodologists we have completed two RCTs in education. We tried to conduct them according to health care standards (i.e., CONSORT). Computers to teach spelling to children; Incentives to retain adult learners in evening classes.
  • Slide 30
  • Computer trial Since 1997 the Government has boasted that > 1.7 billion of taxpayers money has been put into equipping schools with computer technology. Very few RCTs have been undertaken to evaluate and substantiate this investment. We supported the largest RCT of computer technology ever undertaken in the UK.
  • Slide 31
  • Methodological challenges Contamination children in the control group may access the technology. Deliver intervention on a lap top. Secure randomisation. Randomisation done by York Trials Unit series of blocked allocation to match availability of laptops (pre-test data did not inform allocation). Avoiding ascertainment bias. Post-tests given to all children in same room by invigilators blind to allocation, test marked by marker blind to group allocation. Pre-tests done blindly. Resentful demoralisation. Waiting list control control children to get intervention at end of term.
  • Slide 32
  • Methods Based in one school. Lack of funding meant that it couldnt be extended elsewhere. All year 7 pupils were randomised to receive literacy instruction via computer or not. Sample size (n = 157) allowed us to have > 80% power to show 0.5 of effect size between groups. Sample size fixed we couldnt increase it, if we had wanted. Waiting list design all pupils received intervention half at the beginning of term half later in the term (normal practice was to arbitrarily assign pupils).
  • Slide 33
  • Participant Flow in Trial
  • Slide 34
  • Results VariableICT Group Means (SD) No ICT Group Means (SD) Adjusted Difference (Intervention Control) (95% CI) P value of adjusted Difference Spelling38.57 (18.00) 37.24 (18.33) 0.954 (-1.83, 3.74) 0.500 Reading18.97 (6.11) 20.38 (6.64) -2.33 (-0.96, -3.71) 0.001 Brooks et al, Ed Studies 2006
  • Slide 35
  • Computers and Literacy In this study we found no benefit of using computers for teaching literacy skills. A larger recently completed trial in the USA (n = 512) found similar results no benefit. > $90 million had been spent implementing Fast forword before it was found to be ineffective. Rouse et al, 2004; NBER Working Paper Series, 10315.
  • Slide 36
  • Incentives Poor attendance to adult literacy and numeracy classes is a major problem. Recent government policy has offered either financial incentives to attend or financial penalties (reduction in benefit) if attendance is poor. Neither approach has been tested in an RCT. With colleagues in Sheffield we undertook a RCT of incentives.
  • Slide 37
  • Methodological challenges Incentive may lead to resentful demoralisation if we randomised individuals. Cluster or class allocation of 28 classes. Small number of units of allocation may result in chance bias or numerical imbalance. Used a minimisation algorithm to achieve balance on observed variables.
  • Slide 38
  • Design Cluster or class based RCT. 28 classes were randomised to either receive 5 per class attended or no incentive. Outcome was number of sessions attended.
  • Slide 39
  • VariableControl (n = 70) Intervention (n = 82) Mean (SD) number of sessions attended 6.69 (2.71) 5.28 (2.79) Mean (SD) post-test literacy scores 21.14 (8.84) 19.01 (8.68) Significant reduction of about 1.5 sessions (95% confidence interval 0.28, 2.79; p = 0.019) (adjusted for clustering, cluster size and pre-test literacy scores
  • Slide 40
  • Incentives Small 5 incentives are counter productive and reduce attendance. Larger incentives may work but we need to evaluate these in a large RCT with educational achievement as main outcome.
  • Slide 41
  • Differences between health and education Education trials are much easier and quicker. Pre-test variables strongly predict post- test, which usually isnt the case in health care trials, consequently pre-test measurement is particularly useful to gain more statistical power.
  • Slide 42
  • Trial designs Unequal allocation Good to use when resources limit us to the amount of one intervention (e.g., mentoring sessions). Cluster designs Avoiding contamination, care needs to be taken on using ITT and recruitment bias. Factorial designs Two trials for the price of one good value! Balanced design Controls for Hawthorne effect (e.g., Intervention group gets maths, control group gets English and 2 trials for the price of one).
  • Slide 43
  • Factorial Design Computer teachingNo computers and no assistants. Computer teaching Class room assistants
  • Slide 44
  • Challenges Follow-up this is crucial to maximise this. Schools are good children are there and apart from (random?) absences most will be present for post-test Adults may need encouragement (e.g., possible incentives to complete follow-up) or statutory data collection (e.g., arrests) allows ITT to be completed. Convincing other researchers that RCTs are feasible and worth doing.
  • Slide 45
  • A few fallacies We cant accurately measure an important variable (e.g., social class) that really predicts outcome our trial will be biased. No it wont, good measurement of confounders may be helpful but randomisation will cancel their effects out. Non compliance means randomisation is not possible. Poor compliance will dilute the treatment effects, however, using ITT analysis will give a public policy estimate of effect and other statistical techniques (e.g., CACE) may adjust for non-compliance.
  • Slide 46
  • Can trials overcome strong beliefs? Often not. RCT of 25 schools to be given an enhanced sex education programme aim to reduce abortions among teenage women. New package cost 900 per teacher compared with 20 for conventional package.
  • Slide 47
  • Results Termination rate among intervention schools was higher (15.7 per 1,000 women 95% CI -10.7 to 42.1, p = 0.26). Difference not statistically significant, but point estimate is a 14% increase risk of terminations.
  • Slide 48
  • What did the authors conclude? High quality sex education should be continued, but to reduce unwanted pregancies complementary, longer term interventions that address socioeconomic inequalities and the influence of parents should be developed and rigorously evaluated. How does this recommendation derive from the data?
  • Slide 49
  • Summary Trials have challenges some simple mistakes can result in a trial losing its internal validity. Many of the problems of trials can be overcome with careful thought and preparation.