Methods for Cluster Randomized Trialschrp.org/wp-content/uploads/2016/11/ichps2010CRT_final.pdf1. Donner A Klar N (2000) Design and Analysis of Cluster Randomization Trials in Health

Designing Cluster Randomized Trials for Health Policy Research

8th International Conference on Health Policy Statistics [ICHPS]

Washington, DC: January 20, 2010

Thomas E. Love, Ph.D. [email protected] Neal V. Dawson, M.D. [email protected] Randall D. Cebul, M.D. [email protected]

Center for Health Care Research and Policy Case Western Reserve University at MetroHealth Medical Center

Cleveland, Ohio www.chrp.org

Workshop Schedule

3:30 – 3:45 Introduction to EMR-Centered Decision Support and Design of a Cluster Randomized Trial [Love, for Cebul]

3:45 – 4:00 Group Task: Designing a Cluster Randomized Trial

4:00 – 4:15 Discussion of Group Task / Strategic Concerns [Group]

4:15 – 4:40 Ethics and Implementation / “When Stuff Happens” [Dawson]

4:40 – 5:05 A Glimpse at Some Analytic Concerns [Love]

5:05 – 5:15 Additional Group Discussion and Workshop Evaluation

mailto:[email protected]�



http://www.chrp.org/�

8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research

RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page ii

Group Task

Study: Improving the care and outcomes of patients with diabetes

Background: A health care system with a well-established electronic medical record

(EMR) capable of providing various Clinical Decision Support (CDS) functions decides

to undertake a controlled trial to determine whether CDS can improve the care and

outcomes of its approximately 9,000 patients with diabetes. Selected characteristics of

the system are described below, including the proportion of its patients by insurance

class. The system asks for your help in planning and designing the study.

Sites Primary Care Providers Diabetes Patients

10 115 8,804

Primary Insurance Class (%) Commercial / Medicare Medicaid Uninsured / Self-Pay

59 26 15

Task: As a group, you’ll have 15 minutes to select a reporter/scribe, discuss the key

features of the trial that you would recommend they undertake, and then prepare a one-

minute oral report

for the rest of the workshop which describes your group’s views

on the three concerns below, and any additional issues you come up with.

Specifically, it would be helpful to address the following concerns:

a. What would the intervention(s) be (i.e; what would you be comparing)?

b. What would the unit of assignment (to the interventions) be?

c. Identify other information that you would request in order to improve

your recommendations.


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page iii

A Few Terms/Acronyms We May Use

CCHIT www.cchit.org/ Created to establish national standards in ambulatory and inpatient EMRs (functionality, interoperability, and security), CCHIT is a coalition of private sector organizations (the American Health Information Management Association, the Healthcare Information and Management Systems Society and The National Alliance for Health Information Technology; released list of first certified ambulatory EMR systems in July, 2006.

Certification Commission for Healthcare Information Technology.

CDS Clinical Decision Support

. In the context of this course, CDS is understood to mean "real-time" clinical decision support that is integrated with functions of electronic medical records systems

CRT Cluster Randomized Trial

. The subject of this course! In contrast to conventional RCT, in which the unit of assignment is typically the same (e.g., a patient) as the unit of analysis.

CPOE Computerized Physician Order Entry

. A function of electronic medical records systems, may be integrated with clinical decision support functions to improve quality, safety, etc (e.g., warning of potential drug interactions)

ICC Intra-cluster Correlation Coefficient

. Also described as rho (ρ), the ICC is the main metric of "clustering"; the amount of variance in a measure that is attributable to differences between clusters; ranges from 0-1. ICCs near zero imply virtually complete statistical independence of members across clusters in a CRT.

MRC United Kingdom's Medical Research Council

. Established a classification system and made recommendations for types of informed consent in CRTs.


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page iv

A Very Partial Bibliography on Methods for Cluster-Randomized Trials Compiled by TE Love

Textbooks

1. Donner A Klar N (2000) Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold.

2. Murray DM (1998) Design and Analysis of Group-Randomized Trials. New York: Oxford University Press.

3. Cochrane Handbook for Systematic Reviews: Section on Cluster-Randomized Trials. http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter_16/16_3_cluster_randomized_trials.htm

Review and General Articles on CRTs

4. Cebul RD Dawson NV Love TE (2007) Cluster Randomized Trials in Health Care Research. Textbook of Clinical Trials, 2nd Edition, edited by Machin D Day S Green S Chapter 36. Wiley.

5. Campbell MK Elbourne DR Altman DF for the CONSORT group (2004) CONSORT (Consolidated Standards of Reporting Trials) Statement: Extension to cluster randomized trials. BMJ 328: 702-708.

6. Cluster Randomised Trials: Methodological and Ethical Considerations, Clinical Trials Series. London: Medical Research Council (2002).

7. Donner A Klar N (2004) Pitfalls of and controversies in cluster randomization trials. Am J Public Health 94: 416-422.

8. Eldridge SM, Ashby D, Feder GS, Rudnicka AR, Ukoumunne OC. Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials (2004) 1(1): 80–90.

9. Grimshaw JM Eccles M Campbell MK Elbourne D Cluster randomised trials of professional and organizational behaviour change interventions in health care settings. [Online at http://www.campbellcollaboration.org/Bellagio/Papers/crt_grimshaw.pdf]

10. MacLennan GS Ramsay CR Mollison J Campbell M Grimshaw J Thomas R (2003) Room for improvement in the reporting of cluster randomized trials in behavior change research. Controlled Clinical Trials 24: 69S-70S.

11. Murray DM Varnell SP Blitstein JL (2004) Design and analysis of Group-Randomized Trials: A review of recent methodological developments. Am J Public Health 94: 423-432.

12. Puffer S Torgerson D Watson J (2003) Evidence for risk of bias in cluster randomized trials: review of recent trials published in three general medical journals. BMJ 327 (7418): 785-9.

13. Simpson JM, Klar N, Donner A. Accounting for cluster randomization: a review of primary prevention trials, 1990 through 1993. Am J Public Health (1995) 85(10): 1378–83.

http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter_16/16_3_cluster_randomized_trials.htm�

http://www.campbellcollaboration.org/Bellagio/Papers/crt_grimshaw.pdf�


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page v

Designing CRTs: Epidemiologic Concerns (Power, etc.)

14. Campbell MK Mollison J Grimshaw JM (2001) Cluster trials in implementation research: Estimation of intracluster correlation coefficients and sample size. Statistics in Medicine 20: 391-399.

15. Campbell MK Thomson S Ramsay CR MacLennan GS Grimshaw JM (2004) Sample size calculator for cluster randomised trials. Comput Biol Med 34:113-125. [Note that the calculator is found at the University of Aberdeen’s Health Service Research Unit Web Site: http://www.abdn.ac.uk/hsru/epp/cluster.shtml which also contains an ICC database.]

16. Campbell MK, Fayers PM, Grimshaw JM. Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research. Clin Trials (2005) 2(2): 99–107.

17. Campbell MK, Grimshaw JM, Steen N. Sample size calculations for cluster randomised trials. Changing Professional Practice in Europe Group (EU BIOMED II Concerted Action). J Health Serv Res Policy (2000) 5(1): 12–16.

18. Carter BR Hood K (2008) Balance algorithm for cluster randomized trials. BMC Med Res Methodology 8:65 http://www.biomedcentral.com/1471-2288/8/65

19. Eldridge SM Cryer C Feder GS Underwood M (2001) Sample size calculations for intervention trials in primary care randomizing by primary care group: An empirical illustration from one proposed intervention trial. Statistics in Medicine 20: 367-376.

20. Gail MH Mark SD Carroll RJ Green SB Pee D (1996) On design considerations and randomization-based inference for community intervention trials. Statistics in Medicine 15: 1069-1092.

21. Giraudeau B Ravaud P (2009) Preventing bias in cluster randomized trials. PLoS Medicine 6(5): e1000065. http://clinicaltrials.ploshubs.org/article/info:doi%2F10.1371%2Fjournal.pmed.1000065

22. Glynn RJ Brookhart MA Stedman M Avorn J Solomon DH (2007) Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers Med Care 45: S38-S43. Available at http://effectivehealthcare.ahrq.gov/repFiles/MedCare/s38.pdf

23. Guittet L Ravaud P Giraudeau B (2006) Planning a cluster randomized trial with unequal cluster sizes: practical issues involving continuous outcomes. BMC Medical Research Methodology 6:17 http://www.biomedcentral.com/1471-2288/6/17

24. Godwin M Ruhland L Casson I et al. (2003) Pragmatic controlled clinical trials in primary care: The struggle between external and internal validity. BMC Medical Research Methodology 3: 28.

25. Kerry SM Bland JM (1998) Sample size in cluster randomization. BMJ 316: 549. 26. Kerry SM Bland JM (1998) The intracluster correlation coefficient in cluster randomization.

BMJ 316: 1455. 27. Lewsey JD (2004) Comparing completely and stratified randomized designs in cluster

randomized trials when the stratifying factor is cluster size: A simulation study. Statistics in Medicine 23: 897-905.

28. Love TE Cebul RD Einstadter D Jain AK Miller H Harris CM Greco PJ Husak SS Dawson NV Electronic medical record-assisted design of a cluster-randomized trial to improve diabetes care and outcomes. J Gen Internal Med 2008; 23(4): 383-391.

http://www.abdn.ac.uk/hsru/epp/cluster.shtml�

http://www.biomedcentral.com/1471-2288/8/65�

http://clinicaltrials.ploshubs.org/article/info:doi%2F10.1371%2Fjournal.pmed.1000065�

http://effectivehealthcare.ahrq.gov/repFiles/MedCare/s38.pdf�

http://www.biomedcentral.com/1471-2288/6/17�


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page vi

29. Mickey RM Goodwin GD (1993) The magnitude and variability of design effects for community intervention studies. Am J Epidemiol 137: 9-18.

30. Raab GM, Butcher I. Balance in cluster randomized trials. Stat Med (2001) 20(3): 351–65. 31. Raab GM, Butcher I. Randomization inference for balanced cluster-randomized trials. Clin

Trials (2005) 2(2): 130–40. 32. Raudenbush SW (1997) Statistical analysis and optimal design for cluster randomized trials.

Psychological Methods 2, 173-185. [See “Optimal Design” under Sprybook J (2006)] 33. Raudenbush SW Liu X (2000) Statistical power and optimal design for multisite randomized

trials. Psychological Methods 5, 199-213. [See Sprybook J (2006)] 34. Sprybook J Raudenbush SW Liu X Congdon R (2006) Optimal Design for Longitudinal and

Multi-Site Research: Documentation for the “Optimal Design” software. [Documentation and software available at http://sitemaker.umich.edu/group-based/optimal_design_software]

35. Thompson SG Pyke SDM Hardy RJ (1997) The design and analysis of paired cluster randomized trials: An application of meta-analysis techniques. Statistics in Medicine 16: 2063-2079.

Analyzing CRTs: Biostatistical Concerns

36. Berger VW Exner DV (1999) Detecting selection bias in randomized clinical trials. Controlled Clinical Trials 20: 319-327.

37. Borm GF Melis RJF Teerenstra S Peer PG (2005) Pseudo cluster randomization: A treatment allocation method to minimize contamination and selection bias. Statistics in Medicine 24: 3535-3547.

38. Braun TM Feng ZD (2003) Identifying settings when permutation tests have nominal size with paired, binary outcome, group randomized trials. J Nonparametr Stat 15: 653–63.

39. Dobbins TA Simpson JM (2002) Comparison of tests for categorical data from a stratified cluster randomized trial. Statistics in Medicine 21: 3835-3846.

40. Donner A (1987) Statistical methodology for paired cluster designs. Am J Epidemiol 126: 972-979.

41. Donner A Klar N (1993) Confidence interval construction for effect measures arising from cluster randomization trials. J Clin Epidemiol 46: 123-131.

42. Donner A Klar N (1994) Methods for comparing event rates in intervention studies when the unit of allocation is a cluster. Am J Epidemiol 140: 279-289.

43. Giraudeau B (2006) Model misspecification and overestimation of the intraclass correlation coefficient in cluster randomized trials. Statistics in Medicine 25: 957-964.

44. Hansen BB Bowers J (2008) Covariate balance in simple stratified, and clustered comparative studies. Statistical Science 23 (2): 219-236.

45. Kerry SM Bland JM (1998) Analysis of a trial randomized in clusters. BMJ 316: 54. 46. Koepsell TD Martin DC Diehr PH et al. (1991) Data analysis and sample size issues in

evaluations of community-based health promotion and disease prevention programs: A mixed-model analysis of variance approach. J Clin Epidemiol 44: 701-713.

47. Murray DM, Blitstein JL. Methods to reduce the impact of intraclass correlation in group randomized trials. Eval Rev (2003) 27(1): 79–103.

http://sitemaker.umich.edu/group-based/optimal_design_software�

http://sitemaker.umich.edu/group-based/optimal_design_software�


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page vii

48. Nixon RM Thompson SG (2003) Baseline adjustments for binary data in repeated cross-sectional cluster randomized trials. Statistics in Medicine 22: 2673-2692.

49. Peters TJ Richards SH Bankhead CR Ades AE Sterne JAC (2003) Comparison of methods for analyzing cluster randomized trials: An example involving a factorial design. Int J Epidemiol 32: 840-846.

50. Rosenbaum PR. Covariance adjustment in randomized experiments and observational studies. Stat Sci (2002) 17: 286–304.

51. Saha KK Paul SR (2005) Bias-corrected maximum likelihood estimator of the intraclass correlation parameter for binary data. Statistics in Medicine 24: 3497-3512.

52. Spiegelhalter DJ (2001) Bayesian methods for cluster randomized trials with continuous responses. Statistics in Medicine 20: 435-452.

53. Turner RM Omar RZ Thompson SG (2001) Bayesian methods of analysis for cluster randomized trials with binary outcome data. Statistics in Medicine 20: 453-472.

54. Ukoumunne OC Thompson SG (2001) Analysis of cluster randomized trials with repeated cross-sectional binary measurements. Statistics in Medicine 20: 417-433.

55. Yudkin PL Moher M (2001) Putting theory into practice: A cluster randomized trial with a small number of clusters. Statistics in Medicine 20: 341-349.

Ethical and Human Subjects Concerns in CRTs

56. Eldridge SM Ashby D Feder GS (2005) Informed patient consent to participation in cluster randomized trials: An empirical exploration of trials in primary care. Clinical Trials 2: 91-98.

57. Hutton JL (2001) Are distinctive ethical principles required for cluster randomized controlled trials? Statistics in Medicine 20: 473-488.

Information on CRTs and Diabetes, CDS, and Related Fields

58. Bland M (2003) Cluster randomized trials in the medical literature. [Online at http://www.users.york.ac.uk/~mb55/talks/clusml.htm]

59. Borgermans L Goderis G van den Broeke C et al. (2008) A cluster randomized trial to improve adherence to evidence-based guidelines on diabetes and reduce clinical inertia in primary care physicians in Belgium: study protocol. Implementation Science 3: 42. Available online: http://www.implementationscience.com/content/3/1/42

60. Cleveringa FG Gorter KJ van den Donk M Rutten GE (2008) Combined task delegation, computerized decision support and feedback improve cardiovascular risk for type 2 diabetic patients: A cluster randomized trial in primary care. Diabetes Care 31: 2273-2275.

http://www.users.york.ac.uk/~mb55/talks/clusml.htm�

http://www.implementationscience.com/content/3/1/42�


RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page viii

61. Fretheim A Oxman AD Havelsrud K et al. (2006) Rational Prescribing in Primary Care (RaPP): A Cluster Randomized Trial of a Tailored Intervention. PLoS Med 3(6): e134.

62. Glasgow RE Nutting PA King DK et al. (2004) A practical randomized trial to improve diabetes care. J Gen Intern Med 19: 1167-1174.

63. Glasgow RE Nutting PA King DK et al. (2005) Randomized effectiveness trial of a computer-assisted intervention to improve diabetes care. Diabetes Care 28, 33-39.

64. Grant RW Cagliero E Sullivan CM et al. (2004) A controlled trial of population management: diabetes mellitus: putting evidence into practice (DM-PEP). Diabetes Care 27: 2299-2305.

65. Gulmezoglu AM Villar J Grimshaw JK et al. (2004) Cluster randomized trial of an active, multifaceted information dissemination intervention based on the WHO reproductive health library to change obstetric practices: Methods and design issues. BMC Medical Research Methodology 4: 2.

66. Han (2005) Unexpected increased mortality after implementation of a commercially sold CPOE system. Pediatrics 116: 506-12.

67. Hillestad R et al. (2005) Can EMR systems transform health care? Potential health benefits, savings, and costs. Health Aff. 24: 1103-1117.

68. MacLean CD Gagnon M Callas P Littenberg B (2009) The Vermont Diabetes Information system: A Cluster Randomized Trial of a Population Based Decision Support System. JGIM 24 (12). Available online: http://www.springerlink.com/content/j175384173jhx410/fulltext.pdf

69. Meigs JB Cagliero E Dubey A et al. (2003) A controlled trial of web-based diabetes disease management. Diabetes Care 26: 750-757.

70. Montori VM Dinneen SF Gorman CA et al. (2002) The impact of planned care and a diabetes electronic management system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care 25: 1952-1957.

71. Moore H Summerbell CD Vail A Greenwood DC Adamson AJ (2001) The design features and practicalities of conducting a pragmatic cluster randomized trial of obesity management in primary care. Statistics in Medicine 20: 331-340.

72. Parker DR, Evangelou E, Eaton CB. Intraclass correlation coefficients for cluster randomized trials in primary care: the cholesterol education and research trial (CEART). Contemp Clin Trials (2005) 26(2): 260–7.

73. Samore MH Bateman K Alder SC et al. (2005) Clinical decision support and appropriateness of antimicrobial prescribing. JAMA 294: 2305-2314.

74. Sequist TD Gandhi TK Karson AS et al. (2005) A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Amer Med Informatics Assoc 12: 431-437.

75. Slymen DJ Elder JP Litrownik AJ Ayala GX Campbell NR (2003) Some methodologic issues in analyzing data from a randomized adolescent tobacco and alcohol use prevention trial. J Clin Epidemiol 56: 332-340.

76. Sturt J Hearnshaw H Farmer A Dale J Eldridge S (2006) The Diabetes Manual trial protocol – a cluster randomized controlled trial of a self-management intervention for type 2 diabetes. BMC Fam Pract 7:45.

http://www.springerlink.com/content/j175384173jhx410/fulltext.pdf�

8th International Conference on Health Policy Statistics – Washington DC – January 2010

Cluster Randomized Trials in Health Policy Research

Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support

Randell D. Cebul, M. D. [email protected] Part 1: Page 1

Cluster Randomized Trials (CRTs)in Health Policy Research

(with a focus on CRTs of clinical decision support)

Thomas E. Love, PhD filling in for

Randall D. Cebul, MDJanuary 20, 2010

Center for Health Care Research & PolicyCase Western Reserve University at

MetroHealth Medical [email protected]

Cluster Randomized Trials?

As compared to individually randomized trials (RCTs),

• CRTs are more complex to design

2

• CRTs require more participants to obtain equivalent statistical power

• CRTs require more complex analysis

Overview of Workshop

• Introduction and Context– Electronic medical records (EMRs) to provide and examine clinical decision support (CDS)

• Group Task: Design a Study• Group Task: Design a Study• Task Review• CRTs: Why they are necessary and challenging to execute

• Analytic and Strategic Concerns• Discussion





Clinical Decision Support (CDS)

Consensus CDS Definition (AMIA):“Providing clinicians, patients, or individuals with knowledge and person‐specific or population information intelligently filteredpopulation information, intelligently filtered or presented at appropriate times, to foster better health processes, better individual patient care, and better population health.”

Clinical Decision Support, informally

Giving the right informationto the right person

at the right time and placeat the right time and place,

making the right decision the easy decision.

Illustrative RealIllustrative Real‐‐time CDS time CDS ––designed to change (?improve) behaviordesigned to change (?improve) behavior

• Provider centered CDS

• Inpatient • Computerized Physician Order Entry, “Smart tools”, guidelines, etc.

• Outpatient • Alerts, linked orders/referrals, links to patient/MD education, etc.

• Patientcentered• access to webbased records, reminders, input/feedback, etc.





EncounterEncounter‐‐based Alert to Improve care for based Alert to Improve care for Diabetic Patients with Leaking Kidneys Diabetic Patients with Leaking Kidneys

What do we know about this patient?What do we know about this patient?

{Links to Automated Order Set}

• She has diabetes and is visiting her PCP• Her kidneys are leaking protein.• She is not on an ACE inhibitor or ARB and has no documented allergies to them.

• She has no other contraindications (K, Cr)•• There are several alternative drugs/dosesThere are several alternative drugs/doses

Automated Order Set Automated Order Set Linked to ACE/ARB AlertLinked to ACE/ARB Alert

Patient name

Patient name

Re-cap of indications

Choice of Rxs/doses

Follow-up testing

Comparative Performance FeedbackComparative Performance Feedback

“My panel” vs. Comparator





CDS for Patients: Viewing their information,CDS for Patients: Viewing their information,inputting data, getting feedback at homeinputting data, getting feedback at home

CDS with EMRs: Untapped Promise?CDS with EMRs: Untapped Promise?

“we conclude that effective EMR implementation…could save more than $81 billion annually and that HITenabled management of chronic disease could eventually double those savings while increasing health benefits.”

– Hillestad R et.al. (2004)

Or… increased mortality (with CPOE)?Or… increased mortality (with CPOE)?

• Pittsburgh Children’s (Cerner)– 13 months preCPOE– 5 months postCPOE

• 1,942 children admitted for specialty care75 deaths– 75 deaths• 39 / 1394 preCPOE (2.80%)(2.80%)• 36 / 548 post (6.57%)(6.57%)

–– Multivariate OR 3.28 (CI 1.94 Multivariate OR 3.28 (CI 1.94 –– 5.55)5.55)• But: retrospective, short postimplementation…

• Han (2005)





Need for EvidenceNeed for Evidence

• What do we need to know?– Will providers adopt CDS? – Safety of Implementation, System Effects– Safety and Quality Implications of Alerts– FPs, Alert Fatigue– Cost implications

• Will they result in more costeffective care?• What is most effective? Over what time horizon?

– Etc – the possibilities are endless…

• How do we study CDS?

Group TasksGroup Tasks

1. Describe key features of a CDS‐catalyzed trial to improve the care and outcomes of patients with diabetes.

2 Identify other information needed to2. Identify other information needed to improve the trial.

One‐Minute Oral ReportsA health care system with an established electronic medical record (EMR) capable of providing various clinical decision support (CDS) functions decides toundertake a controlled trial to determine whether CDS can improve care and outcomes of its 8804

diabetic patients

a. What would the intervention be?b. What should be the unit of assignment?c. What other information do you want?

diabetic patients. 10 sites, 115 primary care providers, 8804 DM patients

Insurance: 59% commercial or Medicare, 26% Medicaid, 15% Uninsured





Task Review – Part A: InterventionWhat Should We Be Comparing?

Part B: Unit of Assignment?

Task Review –Part C: Other Useful Information

• RCT (Randomized Controlled Trial)– unit that is randomized is the unit of analysis– for large n, important attributes are likely to be distributed similarly across groups

• CRT (Cluster‐Randomized Trial)

Comparing RCTs and CRTs

• CRT (Cluster‐Randomized Trial)– interventions focus on systems, prevention, behavior

– useful when “contamination” is likely to be a problem and/or blinding is impossible

– unit randomized is not the sole unit of interest – “clustering” within units is the critical issue





Patients within Physicians within Practices within Study Groups: A 4‐Level CRT

19

Cebul RD Dawson NV Love TE (2007) Textbook of Clinical Trials, 2nd Ed., Wiley.

CRTs: Impact of Clustering

If there are important cross‐group differences in important factors at baseline, this affects:

• Study Power [Effective Sample Size]

• Study Design: y g– Pre‐randomization cluster “balancing” will improve balance of important prognostic factors

• Study Analysis:– Need analytic techniques that account for clustering: e.g. GEE, hierarchical models, etc.

8th International Conference on Health Policy Statistics – Washington– January 2010


Part Two: Ethics and Implementation of Cluster Randomized Trials

Neal V. Dawson, MD [email protected] Part 2: Page 1

Ethics and Implementation

When CRT = Creative Responses to Trying Circumstances

1

“Stuff Happens”

Neal V. Dawson, MDJanuary 2010

[email protected]

Ethical Issues in CRTs

• Wide spectrum of CRT designs

–Can resemble individual RCTs where each subject decides whether to participate

–May involve the randomization of whole practices, communities, or countries

UK MRC Clinical Trials Series

• Type A trials – are structured such that they do not allow participation decisions by individuals– Often CRTs

• Type B trials – allow individual subjects to decide about participation– Usually individual RCTs





Usual Ethical Concerns Apply

• Potential to produce findings that can improve human health or welfare

• Favorable balance of risks and benefits• Conflicts: subjects’ welfare prevails over Conflicts: subjects welfare prevails over

interests of science and society• Voluntary informed consent when possible

– Alternative safeguards – cluster representation

• Alternative consent mechanisms• Review by independent ethics review

committee

Ethical Concerns for Type A Trials

• Mechanisms for representing the interests of the cluster – Representative (person or group)– Sufficient knowledge of circumstances, beliefs,

and values of cluster– Delegated authority from or for the cluster– No conflicts of interest

• Cluster representative: ‘individual rights’– Suitably informed– Able to withdraw without adverse impact

Specific Trial Examples

• Many quality improvement studies– Interventions on groups rather than individuals

– Interventions targeted at health care professionals

Ad i i t ti t i l• Administrative trials– Studies that do not intrude into physician-patient

decision making

– Studies of aspects of health care activities about which patients never make decisions (e.g., patient scheduling schemes to improve patient flow)





Written Informed Consent• May be impractical and produce important amounts

of bias

– Tu et al. NEJM 2004;350:1414-21

– Attempted written informed consent of consecutive patients for stroke registry in Canadaconsecutive patients for stroke registry in Canada

– Participation: Phase 1=39% (4285 eligible), Phase 2=51% (2823 eligible); Many died or were discharged before they could be approached

– Selection bias: Inhospital mortality, Enrolled=6.9%; Not enrolled=21.7%;

– Registry patients not representative

Alternative Consent Mechanisms

• Goldberg. Medical Care 1990;28:822-33• Administrative trials: Firm trials and many CRTs

where doctor-patient decision making is not compelled or constrained‘P i N ifi i ’ b di h b • ‘Prior Notification’ – about studies that may be done to improve care; similar to commonly used notifications about possible use of patient records for epidemiologic or biomedical research (HIPPA issues are separate)

• “You and your physician will always be able to determine which tests and treatments you will receive.”

Opt-out Strategies

• Mechanism for passive consent• Littenberg and MacLean. JGIM 2006;21:207-11

– Quality improvement intervention: Vermont Diabetes Information SystemMulti state randomized trial– Multi-state randomized trial

– Patients were notified by mail that they were eligible – Were able to opt-out by calling a toll-free number– Of 7558 patients invited to participate, 210 (2.8%) opted-

out

• Recruitment of a ‘broad and representative’ sample– Maintains appropriate protections for study subjects





Implementing CRTs: “Observational Studies”

• Some implementation challenges can make CRTs look like observational studies– Political issues– Temporal trendsp

• Challenges regarding – Randomization– Susceptibility– Performance– Detection– Transfer

10

Randomization

• Not all sites will allow randomization• e.g. they ‘need’ to be in one group or

the other

• Sites may need to be randomized • Sites may need to be randomized together

• May be functionally linked• e.g. several clinicians may work

regularly at 2 sites

11

Susceptibility to Outcome at Baseline

• Differential temporal trends• May threaten comparability across

intervention/non-intervention clusters

• e.g. in a study of patient health behaviors, a contract is lost and many patients with private insurance move to another health system leaving a disproportionate number of uninsured patients at one site

12





Performance

• Intervention fidelity – e.g. after the study starts, a third of

physicians at one intervention site decide th l i h t ti i tthey no longer wish to participate

• Co-interventions– e.g. after the study starts, at only one site a

pharmaceutical company provides at no cost a new efficacious but expensive drug that influences the outcome of interest

13

Detection of the Outcome

• e.g. study of ACE inhibitor use to slow diabetes related renal function decline – after the study starts, one group installs

software that automatically records data that leads to more appropriate performance of the gold standard test; at other sites the recording is dependent on clinicians remembering to order the gold standard test

14

Transfer

• Transfer– Drop outs and crossovers

• Sites

• Subjects within a site

• Statistical comparisons that do not appropriately consider hierarchies and clusters

15





Summary

• When studying interventions designed to affect groups, individual-level RCTs risk contamination and Type II errors.– CRTs are often preferable for interventions to affect CRTs are often preferable for interventions to affect

behavior

– Clustering – differences in subjects across groups –affects power, design, and analyses of CRTs

– EMRs facilitate CRTs – pre-assignment “balancing” of important prognostic attributes.

• Implementing CRTs – “stuff happens” and agile designs and analytic plans are called for...

16

8th International Conference on Health Policy Statistics – Washington DC– January 2010

Designing Cluster Randomized Trials in Health Policy Research

Part Three: A Brief Overview of Some Analytic and Strategic Concerns

Thomas E. Love, Ph. D. [email protected] Part 3: Page 1

Designing Cluster Randomized Trials in Health Policy Research :

A Few Analytic d St t i C

1

and Strategic Concerns

Thomas E. Love, Ph. D.January 2010

[email protected]

Cluster Randomized Trials?

As compared to individually randomized trials (RCTs),

• CRTs are more complex to design

2

p g

• CRTs require more participants to obtain equivalent statistical power

• CRTs require more complex analysis

Key Concerns when Doing CRTs

• Unit of randomization (assignment) is different than the unit of analysis

• Clustering has design (sample size) and

3

analytic implications– Need larger samples than individual RCT

– Need better pre-trial data for balancing

– Need sophisticated statistical methods





The ICC(Intra-Cluster Correlation Coefficient)

Variance in the outcome attributableto differences BETWEEN clusters

TOTAL variance in the outcome=

ICC for an outcome

4

• ICC = 0.01 means 1% of the variance in the outcome is attributable to differences between clusters

• ICC is interpreted as if it were “R2”

Toy Example #1

r 1

Clu

ster

2

n=48, mean = 109.0, sd = 14

n=51, mean = 108.7, sd = 16

Estimated ICC < 0.00001

5

90 100 110 120 130 140 150

Clu

ster

Systolic BP (in mm Hg)

Source SS df MS F PBETWEEN Clusters 1.8 1 1.8 0.01 0.93

WITHIN Clusters 22063 97 227.4

STATA1-way

RandomEffects

LONEWAY

er 1

Clu

ste

r 2

Toy Example #2

n=49, mean = 118.4, sd = 13.2

n=50, mean = 99.4, sd = 9.8

Estimated ICC = 0.569

6

90 100 110 120 130 140 150

Clu

ste

Systolic BP (in mm Hg)

Source SS df MS F PBETWEEN Clusters 8961 1 8961 66.3 < .0001

WITHIN Clusters 13102 97

STATA1-way

RandomEffects

LONEWAY





“Basic” Design Effect Formula

• Average Cluster Size: Mean # of subjects per cluster =

• ICC estimate for relevant outcome

n

ICC

7

Design Effect =

ICCn 11

This is the D. E. for continuous data. For comparing two proportions, there isan additional correction due to Cornfeld (1978). See Campbell (2001).

Design Effect

• Design Effect ≥ 1.0

Total subjects requiredunder CLUSTER randomization

Total subjects requiredunder INDIVIDUAL randomization

=

8

g ≥

– If DE > 1.0, CRT requires more patients than would a RCT with the same power.

• For larger ICC, Design Effect increases– CRT requires increasingly larger n relative

to individually randomized trial (RCT).

A Behavioral Intervention in General Practice to Lower Serum Cholesterol

• Practices randomized into two clusters– Intensive dietary intervention vs usual care

– Suppose we recruit 50 patients per practice

Between practice estimate 2 0 046s

9

– Between practice estimate

– Within practice estimate

• ICC estimate for this outcome …

0.0460.0036

0.046 1.28ICC

Kerry SM Bland JM (1998)

0.046Bs 2 1.28Ws





Intervention to Lower Serum Cholesterol

• Practices randomized into two clusters– With n = 50 pts / practice, and ICC = 0.0036,

design effect is

1 1 1 50 1 .0036 1.17DE n ICC

10

• Individual RCT requires n = 5,364 pts to detect 0.1 mmol/L difference (90% power and α = 0.05). So the CRT needs…


1.17 5364 6276 6300 pts

= 126 clusters of 50 patients each

Detecting a 0.10 mmol/L cholesterol Difference (90% power, = 0.05, same ICC)

# needed with individual randomization (i.e. RCT)

5,364 1.00

ICC Pts / Practice Practices PatientsDesignEffect

11

.0036

10 558 5,580 1.04

25 234 5,850 1.09

50 126 6,300 1.17

100 74 7,400 1.38

500 32 16,000 2.98Kerry SM Bland JM (1998)

Are there fancy, free tools to estimate the sample size in CRTs?

• http://www.abdn.ac.uk/hsru/epp/cluster.shtml

– University of Aberdeen sample size

calculator with instructions

12

– Database of ICCs for use in planning

• http://sitemaker.umich.edu/

group-based/optimal_design_software

– “Optimal Design” from U. of Michigan





Patients within Physicians within Practiceswithin Study Groups: A 4-Level CRT

13

Common CDS Setting:# of Practices (clusters) is fixed# of MDs/Pts per Practice is variable

Do I Have Enough Practices?

• Many studies: 5-10 practices / arm

– ICC estimates vary quite a bit from study to study - consider 95% CI bounds?

DIG IT bl f ll ICC

14

– DIG-IT: reasonable power for small ICCs• System A: 2 study groups of 5 practices

• System B: 3 groups: 4, 6, and 4 practices

– Design Effect increases with larger practice sizes, if ICC stays constant

Designing a CRT: More Patients Per Practice or More Practices?

# of Practicesper Treatment Arm

ICC(rho)

Required # of Patients (Total)

Needed # of Pts/Practice

40.0

0.01

0.02

200

396

∞

50

99

∞

15

0.03 ∞ ∞

100.0

0.01

0.02

0.03

200

248

320

486

20

25

32

49

200.0

0.01

0.02

0.03

200

220

246

278

10

11

13

14





Allocating Practices to Study Groups:The DIG-IT study

• Want minimal differences across study groups on important predictors of:– CDS adoption

16

p– Response to the CDS intervention

• Balancing of practices across study groups (pre-randomization) is critical.– EMR plays a large role here– Most important for small # of practices

EMR-Based Balancing Procedure

• For all feasible clusterings of 10 practices into 2 study groups…– Assemble practice-level clinical and demographic

data from EMR

17

data from EMR

– Identify clusterings which appear to balance an array of baseline characteristics / trends

• (Blinded) consensus as to clustering with best balance. Intervention allocated by coin flip to one study group from that clustering.

Love TE Cebul RD Dawson NV et al. (in press) J Gen Internal Med special issue on Health IT

LDLIncome

VisitsBMI

LDL>=130Hospital12m

ACEARBmalbStatinLDL

CHFCADCVA

RenalPsych

CommInsUnIns

DIG-IT selectionGeographical clustering

Which Table 1 do you want?

18

0 50 100 150 200Absolute Standardized Difference

SlopeA1cA1c>=9A1c<=7LastA1cFemale

Afr-AmerHispanic

InsulinPneumVax

FluVaxSmoker

SBPNoShow

EDAgeLDL





Flu Vaccine

Systolic BP

Last A1c <= 7

Last A1c >= 9

Last A1c

Slope A1c

DIG-IT clustering

336 “clusterings” which split 10 practices into 2 study groups of sizes 4 & 6 or 5 & 5

19

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

Intra-Cluster Correlation Coefficient

Female

African-American

Hispanic

Current Smoker

Insulin

ED visit

No Show

Pn Vaccine

Pre-Trial Balancing of Practices Using EMRs

• Result: 5 intervention practices and 5 “usual care” practices with excellent balance across study groups.

20

• EMRs provide new opportunities for state-of-the-art study design. – Could use this approach to create study

groups for lots of community-based therapeutic or health care delivery trials.

ANALYSIS: Problems with Individual Level Analysis of Clustered Data

• Lack of independence among members of a cluster (cluster effect)– Need for larger sample sizes: Standard

sample size formulas will lead to

21

sample size formulas will lead to underpowered studies

– Need for sophisticated statistical methods: Standard approaches will tend to bias p-values downward risking spurious claims of statistical significance

Wojdyla D (2005) Cluster Randomized Trials and Equivalence Trials. WHO available online.





Analyzing a Cluster Randomized TrialThe Simple Approach

• Construct a summary statistic for each cluster, then analyze these summary values– As in repeated measures or meta-analyses– Simple, but doesn’t allow for covariates

22

• Example: 34 practices referring patients to St George’s Hospital for X-ray exams – 17 practices got nothing, 17 got a one-page

laminated copy of referral guidelines

• Outcome: % of x-rays requested which conformed to the guidelines


The Wrong Approach• Act as if we had randomly assigned

individual patients to intervention groups• Calculate difference in proportion of requests

in each group that conform.

Group Total Requests % Conforming

23

p q g

Intervention 429 79.5

Control 702 72.5

Difference is 7.0 % points, with SE of 2.6

95% CI is (2, 12) percentage points, P value = 0.0008 (χ2 test)

WRONG!!

This is just wrong, despite what you see in many meta-analyses

X-Ray Requests Conforming to

Guidelines

Two-Sample TMean (%C) Int. = 81.6

Mean (%C) Ctl. = 73.6

SE (diff) = 4.3, 32 df

Intervention Control

ID Req’s % Conf. Req’s % Conf.1 20 100 7 100

2 7 100 37 89

3 16 94 38 84

4 31 90 28 82

5 20 90 20 80

6 24 88 19 79

7 7 86 9 78

24

( ) ,

95% CI: (-1, 17)percentage points

P value = 0.07Weights each practice

equally

8 6 83 25 76

9 30 83 120 75

10 66 80 88 73

11 5 80 22 68

12 43 77 76 68

13 43 74 21 67

14 23 70 126 66

15 64 69 22 64

16 6 67 34 62

17 18 56 10 40





X-Ray Requests Conforming to

Guidelines

Two-Sample TEstimated

Mean diff = 7.0SE = 3 3

Intervention Control

ID Req’s % Conf. Req’s % Conf.1 20 100 7 100

2 7 100 37 89

3 16 94 38 84

4 31 90 28 82

5 20 90 20 80

6 24 88 19 79

7 7 86 9 78

25

SE = 3.3

95% CI: (0.2, 14)percentage points

P value = 0.04Weights practices

by # requests

8 6 83 25 76

9 30 83 120 75

10 66 80 88 73

11 5 80 22 68

12 43 77 76 68

13 43 74 21 67

14 23 70 126 66

15 64 69 22 64

16 6 67 34 62

17 18 56 10 40

X-Ray Request ResultsMethod 95% CI P value

WRONG – Treat as if individual RCT

(2, 12) 0.008

CRT – T test (equal practice weights)

(0.2, 14) 0.07

26

• Ignoring clustering results in CIs which are too narrow and P values which are too small.

• Reported ICC in this trial turned out to be 0.019

practice weights)

CRT – T test (weight by # of requests)

(1, 17) 0.04


Other Analytic Approaches

• Adjust standard errors using the design effect – an approximation

• Robust variance estimates

GEE

27

• GEE

• Multi-level modeling

• Bayesian hierarchical models

• And much, much more …





Eight Methods Of Analyzing A CRTCRT of 2 interventions designed to increase

breast screening attendance.Outcome: log (OR) of attendance for two

intervention effects and their interaction

28

Three cluster-level analyses1. Unweighted regression of practice log odds

2. Log odds Regression weighted by inverse variance

3. Random-effects meta-regression of log odds with practice as a random effect

Peters TJ et al. (2003)

Eight Methods Of Analyzing A CRT

CRT of 2 interventions designed to increase

breast screening attendance.

Five individual-level analyses

4 S d d l i i i (i l i )

29

4. Standard logistic regression (ignore clustering)

5. Logistic regression: robust standard errors

6. Generalized Estimating Equations

7. Random-Effects logistic regression

8. Bayesian random-effects logistic regression

Peters TJ et al. (2003)

Results of the Eight Analyses

• [4] was highly anti-conservative (i.e. wrong).

• The other (more valid) methods showed …– Large differences in parameter estimates

– Large differences in standard errors

30

Large differences in standard errors

– Some weren’t computationally stable

– Some were more sensitive than others to between-cluster variation

– GEE doesn’t work well with small # of clusters

• More Guidance Is Needed!





Additional Reporting Requirements for CRTs (CONSORT Statement)

1. Rationale for adopting a cluster design

2. How clustering was incorporated into the sample size calculations (include ICCs)

31

3. How clustering effects were incorporated into the analysis

4. The flow of both clusters and individuals through the trial, from assignment to analysis.

Campbell MK et al. for the CONSORT Group (2004)

Repeating Myself:Key Concerns when Doing CRTs

• Unit of randomization (assignment) is different than the unit of analysis

• Clustering has design (sample size) and

32

analytic implications– Need larger samples than individual RCT

– Need better pre-trial data for balancing

– Need sophisticated statistical methods

Documents

Methods for Cluster Randomized Trialschrp.org/wp-content/uploads/2016/11/ichps2010CRT_final.pdf1. Donner A Klar N (2000) Design and Analysis of Cluster Randomization Trials in Health