COMMENTARY

Drug safety studies and measures of effect using the self-controlledcase series design

Kumanan Wilson1,2* and Steven Hawken1

1ICES@Uottawa, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada2Department of Medicine, Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada

The self-controlled case series (SCCS) design hasemerged as an effective methodology for evaluatingdrug safety1. Unlike more traditional observationalstudy designs such as cohort studies or case–controlstudies, the SCCS only requires information on caseswho have received the exposure of interest. Themethodology is especially useful for situations wherethere are marked differences between exposed andunexposed individuals. This is a feature of manyvaccine safety studies, and the SCCS design has,consequently, become a primary methodology for theassessment of vaccine safety2,3.The results of SCCS studies are often presented as

relative incidences. However, in a similar manner torelative risks, these may create the misleading percep-tion of a higher absolute risk from the drug than actu-ally exists because of the very low baseline risk ofmany of these events. In the case of vaccine safety,this is particularly important because misinterpreta-tions by the general public of the absolute risk avaccine poses to an individual may create unnecessaryfear and perpetuate vaccine refusal. We describe amethod for presenting information on risk and somemechanisms by which SCCS data can be used tocalculate number needed to harm.

MEASURES OF EFFECT

In general two different measures of effect are used instudies of harm: relative measures and absolutemeasures. Relative estimates of effect size are believed

to be stable if the baseline risk changes unless theunderlying pathophysiology of the phenomenonchanges. The relative risk reduction associated withremoving an exposure is the simple difference inevents rates between exposed (Pe) and unexposed (Pc)groups divided by the control event rate (Pe-Pc)/Pc.Absolute measures of harm, conversely, are depen-dent on the baseline estimates of risk. The absoluterisk reduction is the simple difference in event rates(Pe-Pc)4. The number needed to treat is another wayof representing absolute measures of harm that may bemore interpretable to patients and is simply the inverseof the absolute risk reduction (1/ARR)5.For low incidence phenomena, relative risks can be

markedly greater than absolute risks. To illustrate,consider the case of ovarian cancer, which has anincidence of approximately 1 in 10 000 (0.0001) in theUnited States6. If exposure to a risk factor increasedthis risk by 10-fold, the relative risk would be 10, andthe relative risk increase would be 900%. However, theabsolute increase in risk would only be 0.0009 or0.09% and the number needed to harm would be 1111.Presenting the risk of an exposure as a 10-fold increasein harm as compared to stating that exposure to the riskcauses harm in less than one out of every 1000 exposedwould likely create different perceptions of risk amongstmembers of the public.

The SCCS design and measures of effect

In studies of vaccine risk, there can be importantconsequences to the choice of which measure ofeffectiveness to use. Given that most adverse events po-tentially associated with vaccination are rare (forexample Guillain-Barré syndrome with the influenzavaccine and thrombocytopenia with the MMR vaccine),

*Correspondence to: Kumanan Wilson, Ottawa Hospital, Civic Campus, 1053Carling Avenue, Administrative Services Building, Room 1009, Box 684,Ottawa, ON K1Y 4E9. E–mail: kwilson@ohri.ca

Copyright © 2012 John Wiley & Sons, Ltd.

pharmacoepidemiology and drug safety 2012Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3337

there could be important impacts on public perception ofvaccine safety when determining if risk should be pre-sented in relative versus absolute terms7,8. Other literaturehas shown that the manner in which measures of effectare presented can influence adoption of an intervention9.The SCCS design has emerged as a method of choice

for evaluating vaccine safety and is also increasinglybeing used to evaluate the safety of other pharmaceuti-cals1. The SCCS design only requires information onindividuals who are both exposed and who experiencethe outcome of interest (cases). By using only cases,and comparing exposed and unexposed periods withinindividuals, the study design implicitly controls for allfixed covariates. This is particularly important in studiesof vaccination safety where unvaccinated individualsdiffer from vaccinated individuals in ways that coulddistort the relationship between the exposure and theoutcome. Another advantage of the SCCS design overcohort or case–control studies for evaluating vaccinesafety is that the latter designs are frequently underpow-ered because the population of unvaccinated children isgenerally very small3. Since the SCCS design onlyrequires data on vaccinated children, it is not affectedby this potential limitation.In the SCCS design, measures of effect are presented

in relative terms. The incidence of an event during anat-risk (exposed) period following exposure is comparedto the incidence of the event in a control (unexposed)period where the exposure could not plausibly have beencausally related to the event. The ratio of the incidence inrisk periods to the incidence in control periods yields arelative incidence. To illustrate, we will use a simpleexample employing the SCCS with a single risk periodof fixed duration, no repeated exposures, and a singlecontrol period with no adjustment for age or seasonalconfounders. In a previous study, we examined the riskof emergency room visits or hospital admissions follow-ing vaccinations in 12 and 18 month old infants inOntario– which included the first and second doses ofthe MMR vaccine10. This population-based study wascarried out using health administrative data and included271 495 vaccinated children at 12 months and 184 312vaccinated children at 18 months. For the 12 monthvaccination, we defined the at-risk period as 4–12 daysafter vaccination, and the control period as 20 to 28 daysafter vaccination. We identified 6462 events, primarilyemergency room visits, during the risk period of 9 daysand 4845 events in the control period of 9 days. Thepoint estimate of relative incidence of events was1.3338 ((6462/9 days)/(4845/9 days)). For typicallymore complex SCCS models, the relative incidenceand its standard error would be obtained using parameterestimates from the fitted model, which can take into

account adjustments for confounders, multiple expo-sures and risk and control periods of varying length.To calculate the excess events due to the exposure, it

is necessary to first calculate the percent of eventsattributable to the exposure (the attributable risk percent)and then multiply this by the number of events inthe risk period2. This attributable risk percent can becalculated using the standard formula (relative risk-1)/relative risk but instead of relative risk, relativeincidence values derived from the model-based estimateare used. The excess events due to the exposure wouldthus be calculated using the formula:

excess eventsdue to exposure ¼

relative incidence� 1ð Þrelative incidence

� �

� of events in risk periodð ÞTo obtain the number of exposures necessary to

cause one excess event, the total number exposed (inthis case vaccinated) should then be divided by theexcess events due to the exposure:

exposures necessary tocause 1 excess event ¼ total number exposed

excess events due to exposure

� �

Adverse event rates in vaccine safety studies are oftenexpressed as excess events per 100 000 vaccinated. Thiscan be calculated by dividing 100 000 by the number ofexposures necessary to cause one excess event:

excess eventsper 100 000 exposed

¼ 100 000exposures necessary tocause 1 excess event

0BB@

1CCA

For vaccination at 12 months, the percentage ofadverse events due to the vaccine using this methodwas [(1.3338-1)/1.3338] * 6462 events in the risk periodor 1617 events. Since 271 495 children were vaccinated,the number of vaccinations needed to produce oneexcess event would be 271 495 vaccinations/1617excess events or one excess adverse event for every168 children vaccinated (Table 1). For vaccination at18 months, the time of the second dose of the MMRvaccine, the risk (exposed) period was considerablyshorter at 3 days during which 1275 children experi-enced at least one event. The control (unexposed) periodremained 9 days during which 3065 children experi-enced at least one event. The relative incidence of eventswas 1.248 which, using the excess event calculator,translated into 1 extra event for every 727 childrenvaccinated (Table 1). While the relative incidence was

k. wilson and s. hawken

Copyright © 2012 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2012DOI: 10.1002/pds

not that much lower at 18 months than at 12 months, themarkedly increased number of children needed to bevaccinated to produce one event was a consequence ofthe considerably shorter period at risk (3 versus 9 days)and thus reduced number of events during the risk pe-riod. This example illustrates why absolute measuresare important to present in SCCS studies as otherwisethe inference would be that there is not much of a differ-ence in effect between the 12 and 18 month vaccinationsince the relative incidences are similar. In fact, the rel-ative incidence rate could have been the same or evenmarginally higher at 18 months and still yield highernumber needed to vaccinate at 18 months compared to12 months because of the shorter risk period at 18months.Using the attributable risk method, it is essential to

recognize that the total number of events for vaccinatedcases in the exposed (risk) period must be known, whichis dependent on both the rate of events in the risk periodand the duration of the risk period, i.e the duration oftime that an individual remains at risk of developingthe adverse event following the exposure. To calculatethe number needed to cause one excess event, thenumber of exposures in the population (the numberof total children vaccinated, not just those who experi-enced adverse events), is also required. The numberneeded to cause an excess event is both sensitive tothe number of events in the risk period and the totalnumber exposed in the population. As the number ofevents in the risk period increases, repeating the calcula-tion will produce a larger number of excess events and

consequently a smaller number needed to be exposedto produce one event, assuming the relative incidenceremains the same. Thus, the reporting of excess eventsor number needed to expose to produce one eventshould be provided in the context of the control and riskperiod event rate and the duration that an individual is atrisk following the exposure.

CONCLUSION

For presenting the results of drug safety studies usingthe SCCS model, we recommend presenting both therelative incidence with confidence intervals and anabsolute measure of harm, either the number needed toexpose to produce one event or the number of eventscaused by 100 000 exposures. This is particularlyimportant for presenting information on harm whenthe baseline risk is very low or the period of risk fordeveloping an adverse event following the exposureis short.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

ACKNOWLEDGEMENTS

No funding was received for this manuscript.

REFERENCES

1. Whitaker HJ, Farrington CP, Spiessens B, Musonda P. Tutorial in biostatistics:the self-controlled case series method. Stat Med 2006; 25(10): 1768–97.

2. Farrington P, Pugh S, Colville A, et al. A new method for active surveillance ofadverse events from diphtheria/tetanus/pertussis and measles/mumps/rubellavaccines. Lancet 1995; 345(8949): 567–9.

3. Farrington CP, Nash J, Miller E. Case series analysis of adverse reactions tovaccines: a comparative evaluation. Am J Epidemiol 1996; 143(11): 1165–73.

4. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure oftreatment effect. Br Med J 1995; 310(6977): 452–4.

5. Laupacis A, Sackett DL, Roberts RS. An assessment of clinically useful mea-sures of the consequences of treatment. N Engl J Med 1988; 318(26): 1728–33.

6. SEER Stat Fact Sheets: Ovary. National Cancer Institute. U.S. National Institutes ofHealth. http://seer.cancer.gov/statfacts/html/ovary.html#referencesSEER (accessedDecember 24, 2011).

7. Lasky T, Terracciano GJ, Magder L, et al. The Guillain-Barre syndrome and the1992–1993 and 1993–1994 influenza vaccines. N Engl J Med 1998; 339(25):1797–802.

8. Mantadakis E, Farmaki E, Buchanan GR. Thrombocytopenic purpura after mea-sles-mumps-rubella vaccination: a systematic review of the literature and guid-ance for management. J Pediatr 156(4): 623–8.

9. Hux JE, Naylor CD. Communicating the benefits of chronic preventive therapy:does the format of efficacy data determine patients’ acceptance of treatment?Med Decis Making 1995; 15(2): 152–7.

10. Wilson K, Hawken S, Kwong JC, et al. Adverse Events following 12 and 18Month Vaccinations: a Population-Based, Self-Controlled Case Series Analysis.PLoS One 2011; 6(12): e27897.

Table 1. Excess event calculator

1. Attributable risk percent =(Relative incidence-1)Relative incidence

2. Excess events due to exposure =(Relative incidence-1) * # events in the risk periodRelative incidence

3. Number needed to expose to cause one excess event =# exposures

(Relative incidence-1) * # events in risk periodRelative incidence12 month vaccination example: 271 495 vaccinated children/[(1.3338-1)/1.3338] * 6462 events in the risk period = 16818 month vaccination example: 184 312 vaccinated children/((1.248-1)/1.248) * 1275 events in the risk period = 727

4. Excess events per 100 000 exposed =100 000

exposures necessary to cause 1 excess event

measures of effect using the self-controlled case series design

Copyright © 2012 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2012DOI: 10.1002/pds