Field evaluation of - MMV

Bulletin of the World Health Organization, 63 (6): 1055-1068 (1985) © World Health Organization 1985

Field evaluation of vaccine efficacy*

WALTER A. ORENSTEIN,' ROGER H. BERNIER,' TIMOTHY J. DONDERO, ALAN R. HINMAN,'JAMES S. MARKS,3 KENNETH J. BART,' & BARRY SIROTKIN'

This paper describes the epidemiological techniques available for measuring vaccineefficacy and recommends a practical approach to their use. Many of the examples relate tomeasles vaccine, the efficacy of which was tested by the techniques described, although themethods are applicable to other vaccines as well. The main advantages anddisadvantages ofthe techniques are indicated.

FIELD EVALUATION OF VACCINE EFFICACY

The effectiveness of a vaccine to prevent diseasedepends on the vaccine being potent and on its properadministration to individuals capable of responding.Useful techniques are available to test the potency ofvaccines and the response of the host. Potency testingis important in monitoring the production of vaccines(to maintain stipulated standards), and theirtransport through the "cold chain". In the latterinstance, vaccines from the field are retrieved andtested to ensure that they have not lost theirpotency.

Serological studies can also be used to determine avaccine's efficacy (1). Seroconversion is useful tomeasure the induction of an immune response in thehost and, in the absence of disease, indicates thepersistence of antibody and immunity. Before begin-ning an immunization programme, these studies canhelp in identifying appropriate target groups forvaccination. Seroprevalence studies monitor theprevalence of antibodies due to disease in the popu-lation and indicate the pattern of occurrence ofdisease. Seroconversion studies are particularlyuseful in choosing the appropriate age for vaccination(2). By knowing the age distribution of measles casesand age-specific seroconversion rates, for example,estimates of the number of preventable cases usingdifferent vaccine policies can be derived (3).

These two techniques (vaccine potency testing and

* Requests for reprints should be addressed to Dr W. A.Orenstein, Technical Information Services, Center for PreventionServices, Centers for Disease Control (CDC), Atlanta, GA 30333,USA.

' Division of Immunization, Center for Prevention Services,CDC.

2 International Health Program Office, CDC.3 Division of Nutrition, Center for Health Promotion and

Education, CDC.

serological testing) can play useful roles in theestablishment and execution of immunization pro-grammes. However, since these procedures dependon laboratory support and may be expensive, it is notfeasible to carry them out under all circumstances.Moreover, the use of seroconversion as an indicatorof vaccine efficacy only measures the efficacy underrelatively controlled conditions during a short period,since pre- and post-immunization sera must be col-lected and the vaccinator is aware that these tests arebeing done. This may not be possible under fieldconditions, such as in an integrated immunizationprogramme where many different immunizationcentres and vaccinators are involved.The success of vaccinations performed under field

conditions can be realistically assessed by measuringthe protection against the disease by epidemiologicalmeans. This can be done without laboratory support.Because of the ease of this technique it can be veryuseful, particularly when the disease occurs invaccinated individuals and there is doubt about theeffectiveness of the vaccination programme. Thisproblem becomes more prominent as vaccinecoverage increases, because there will be more casesof illness occurring in vaccinated persons, even whenthe vaccine efficacy is high (4). When the vaccineefficacy is found to be lower than expected, detailedinvestigations should be carried out to determine thecauses and take corrective action.The techniques described below have been used in

the field evaluation of measles vaccine efficacy, butthey are applicable to other vaccines as well. For thoseinterested in detailed aspects of the methodology,particularly regarding potential biases and how theymay be anticipated and corrected, a more detailedreport has been published by the WHO ExpandedProgramme on Immunization.'

a The field evaluation of vaccine efficacy (unpublished WHOdocument EPI/GEN/84/10).

4617 1055-

1056 W. A. ORENSTEIN ET AL.

CALCULATION OF VACCINE EFFICACY:GENERAL PRINCIPLES

Vaccine efficacy is measured by calculating theincidence rates (attack rates) of disease amongvaccinated and unvaccinated persons and determin-ing the percentage reduction in the incidence rate ofdisease among vaccinated persons compared to un-vaccinated persons. The basic formula is:

VE=ARU-ARV x 100ARU

where VE = vaccine efficacy, ARU = attack rate inthe unvaccinated population, and ARV = attack ratein the vaccinated population.For example, if the vaccine were totally effective,

there would be no disease in the vaccinated popu-lation and the calculation would simplify to:

ARU-0x 100= 1000°ARUxlOl%

By contrast, if the vaccine had no effect at all, ARUwould equal ARV and the calculation would simplifyto:

ARU xlO0%In practice, vaccines are neither perfectly effective

nor totally ineffective. Measles vaccine, for example,is 80-95% effective when appropriately administered(1, 5-8).The ideal vaccine efficacy study is a clinical trial

starting with persons susceptible to disease. In adouble-blind randomized placebo controlled trial,half the subjects receive vaccine and half receiveplacebo. To calculate the vaccine efficacy, bothgroups are followed prospectively to determine theattack rates for disease in vaccinees and non-vaccinees. This type of study is generally not possibleafter a vaccine has been licensed because, when thevaccine is of proven benefit, the use of a placebo isunethical. In most countries today, measles vaccinehas been used in a proportion of the population.These vaccinees are not a randomly selected groupand their susceptibility prior to vaccination is oftenunknown. None the less, vaccine efficacy studies arestill possible by reducing biases to a minimum andrecreating as closely as possible the "ideal"conditions of the prospective clinical trial.Four major factors affect most epidemiological

studies of vaccine efficacy.

(1) Case definition. It is important that a uniformdefinition of cases should be developed and applied toall individuals in the study. This definition should beas specific as possible. Laboratory confirmation of at

least some cases can help demonstrate the accuracy ofthe case definition. A useful clinical case definitionfor measles is: an illness with a generalized rash of 3 ormore days duration, fever () 38.3 °C), and any oneof the following -cough, coryza, or conjunctivitis.bWhen all three criteria are met, the illness is likely tobe measles.

(2) Case ascertainment (case detection). It isimportant to ensure that efforts to detect cases amongvaccinated and unvaccinated populations are equal.Surveillance surveys of the total population, in whichinvestigators go from door to door using a clinicalcase definition to find cases, give the least biasedestimate of vaccine efficacy.

(3) Vaccination status determination. Vaccinationstatus must be determined accurately and, wheneverpossible, based on a recorded date of vaccination. Ifthe records of many vaccinees are lacking, the vaccineefficacy calculations may be biased. Definitions ofvaccination status will depend on the actual type ofinvestigation used. In general, persons can beconsidered as vaccinated against measles if theyreceived vaccine on or after the minimum recom-mended age for vaccination and at least 14 days priorto the onset of disease or of an outbreak. Persons whoreceived vaccine prior to the recommended age shouldnot be classified as unvaccinated but should beclassified in a separate category. Persons vaccinatedin control clinics during an outbreak should beclassified according to their vaccination status priorto the outbreak. Persons of unknown vaccinationstatus or with an incomplete series of vaccinationsshould be excluded from the calculations.

(4) Comparability of exposure. Vaccine efficacyshould be measured under conditions where bothvaccinees and non-vaccinees have an equal chance ofexposure to measles. This is most likely to be the casewhen the incidence rate of the disease is relativelyhigh.

SPECIFIC METHODS

ScreeningA preliminary estimate can easily indicate whether

vaccine efficacy is within expected limits. Formeasles, an attack rate of greater than 10% invaccinated individuals immediately suggests the needfor further evaluation since maximum efficacy will beless than 90%. (Under conditions of 90% efficacy,10% of the vaccinated population is susceptible;

b Provisional guidelinesfor the diagnosis and classification oftheEPI target diseasesfor primary health care, surveillance and specialstudies (unpublished WHO document EPI/GEN/83/4).


therefore less than 100o of persons would be expectedto become ill). However, an attack rate of < 10%among vaccinated persons by itself does not mean thevaccine is effective; comparison with attack rates inthe unvaccinated group is necessary.

In many situations, attack rates among the vac-cinated and unvaccinated will not be known withprecision. However, vaccine efficacy can be estimatedfrom other available information. The vaccineefficacy equation has been formulated as follows:

PCV = PPV - (PPV x VE)1- (PPV x VE)

where PCV = the proportion of cases occurring invaccinated individuals; PPV = the proportion of thepopulation vaccinated; and VE = vaccine efficacy(J.M. Kobayoshi & J.P. Brennan, personal com-munication, 1980). When any two of the threevariables are known, the third can be calculated (4).

Fig. 1 shows curves generated from this equation.These curves indicate the theoretical proportion ofcases that will have a vaccination history in a givensetting for specified levels of vaccine efficacy. Thecurves do not predict the occurrence of an outbreak inany given set of circumstances, but do show theexpected proportional distribution of cases byvaccination status if an outbreak should occur. Byknowing or estimating the proportion of casesoccurring in vaccinated persons and the proportion ofthe population at risk that is vaccinated, an estimate

100PPV-(PPV X

100

1-(PPV X VE) ___A l-- <~~~~~~~IZAII 80

4~~/ y/ 70

- I-I-

I -I-20

- -.~~~ -. / "I--- 30

inI L L _2b 3b 4b sb 6b 7b Ob 9b

J O

00

PPV

Fig. 1. The relationship between the percentage of casesvaccinated (PCV) and the percentage of the populationvaccinated (PPV) for seven different percentage valuesof vaccine efficacy (VE).

of vaccine efficacy can be made.This screening technique will indicate whether there

is need for more careful evaluation. It should not berelied upon for precise estimates of vaccine efficacy.There is a small danger that vaccine efficacy may beoverestimated if the vaccination levels in thecommunity are overestimated (particularly when thevaccine coverage is > 80%o), or if the proportion ofcases with a vaccination history is underestimated.However, in most circumstances with reasonablyaccurate estimates, an overestimation of vaccineefficacy should be rare and this screening will providea rough guide as to whether further evaluation isnecessary.

Outbreak investigations: community-wide or totalpopulation assessment

Criteriaforselection. The best situation in which tomeasure vaccine efficacy is probably in outbreaks indefined settings such as villages, towns, cities orschools (5, 7, 8). Although any outbreak can beinvestigated, biases will be minimized if the followingcriteria are kept in mind: (1) absence of substantialprior disease activity in the studied age group, (2) bothvaccinees and nonvaccinees included in the studypopulation, (3) adequate numbers in the populationin the age group to be studied, (4) high overall attackrate (for measles, generally in excess of 5% in thechosen age group), and (5) good vaccination recordsavailable to differentiate non-vaccinees fromvaccinees.

Efficacy is probably best measured in settingswhere measles has been intermittent rather thanendemic. Generally the selected study group shouldbe old enough to be susceptible to measles and to havebeen vaccinated, yet young enough not to have hadsubstantial exposure to measles prior to the outbreak.In villages, the appropriate age group to be studiedmay be determined by questioning village eldersabout the time of the last outbreak and by deter-mining the proportion of each age group with ahistory of disease before the outbreak. Age groups in-which substantial proportions of persons have hadthe disease should not be included in the study.The lower age limit is determined by the age

recommended for vaccination (2, 3). This is usuallyafter the transplacentally derived immunity haswaned and before the attack rates for the diseasebecome high. For measles, in most developing areas,this age group probably includes children between 9months and 3 years old. If the overall attack rates inthe selected age groups are in excess of 5%, the risk ofexposure may be considered to be somewhatcomparable. Ideally, good vaccination recordsshould be available.

80

70

60

u 50

40

30

20

10

0

1057

II


Methods. The following should be taken intoconsideration.

(1) Case definition: a sensitive and specific clinicalcase definition should be used. The definition givenabove may be used to classify cases of measles.

(2) Case ascertainment: a total population basedsurveillance survey should be conducted. In villages,workers should go from door to door taking a censusof all persons in the target age group and determiningwhether any have had an illness clinically compatiblewith measles. In the case of persons in the target agegroup who have died (from whatever cause) duringthe outbreak period, details of the clinical illness andvaccination status should be collected; these personsshould be included as cases or non-cases, as appro-priate.(3) Vaccination status determination: as the healthworkers go from door to door, they should obtainhistories of vaccination for all persons in the targetage group. Dates of birth and of vaccination shouldbe recorded, if available. If the latter are not avail-able, the age at vaccination should be estimated.Equal effort should be made to determine the vacci-nation status of cases and non-cases. For measles,persons should be considered as immunized if theyreceived the vaccine at or after the minimum recom-mended age, and if vaccinated 14 or more days beforethe onset of the outbreak. Persons vaccinated beforethe recommended age should be classified separately.Persons vaccinated during the outbreak should beclassified based on vaccination status prior to theoutbreak.

(4) Prior disease: disease that occurred before theoutbreak will have minimal effect on the vaccineefficacy calculation if the incidence rate of the diseasein the area under study and in the chosen age groupwas low.' Hence, age groups that probably will havebeen heavily exposed to measles prior to the outbreak(e.g., persons > 3 years of age) should be excludedfrom the investigations. However, once the appro-priate age group is selected, persons with prior diseaseshould be included in the denominators of the appro-priate attack rate calculation. The numerator,however, should include only cases that occurredduring the outbreak.

Analyses (see Table 1). These must take intoaccount the following:

(1) ARU = b+e

c The effects of disease prior to the outbreak will also beminimized if persons with prior disease were as likely to obtainimmunization as persons without disease and the risk of exposure tomeasles of vaccinees and non-vaccinees prior to the outbreak wassimilar.

where b is the number of unvaccinated cases duringthe outbreak and e is the number of unvaccinated whodid not develop measles during the outbreak.

(2) ARV = a

a+d

where a is the number of vaccinated cases duringthe outbreak and d is the number of vaccinated whodid not develop measles during the outbreak.(3) Persons with unknown vaccination historiesshould be excluded from the calculation, whether ornot they had illness (excluding c +f, see Table 1).

(4) Vaccine efficacy can be calculated according tothe standard formula.

Special analyses to evaluate efficacy by the ageat vaccination, the duration of vaccine-inducedimmunity, and the effects of two doses can also becalculated using the ARV for the specific group beingstudied. For example, to calculate vaccine efficacy forpersons vaccinated at exactly 12 months of age, theARV would equal the number of cases who had beenvaccinated at this age divided by the total populationvaccinated at the same age. The overall ARU isusually used for each of the special analyses; morerefined estimates may be obtained by controlling forfactors such as age. In the above example, the ARUmight be calculated only among children aged > 12months at the time of the outbreak.Measurement of vaccine efficacy in outbreaks can

be complicated if an extensive vaccination pro-gramme was carried out during the outbreak becausethe vaccination status of individuals may changeduring the middle of the outbreak. Correction for theeffects of control programmes is necessary if asubstantial proportion of the cases occur after theprogramme and a substantial proportion of the popu-lation was vaccinated during the programme (7, 9).dWhen either of these proportions is small, no cor-rection is necessary.

Outbreak investigations: estimating vaccine efficacyin large populations (cluster samples)When outbreaks occur in large populations, deter-

mination of the vaccination status of all theindividuals involved may be unmanageable. In thesesituations, a coverage survey of children in the at-riskpopulation can be used to estimate the pre-outbreakimmunization levels. For example, thirty neighbour-hood clusters may be chosen, as already described andseven or more children in the age group for the study

d MARKS, J. S. ET AL. A new stochastic method for theepidemiologic evaluation of vaccine efficacy. Paper presented at theRobert Wood Johnson Clinical Scholars Program Meeting,Scottsdale, Arizona, 12-15 November 1980.


Table 1. Data to be collected in an outbreak investi-gation for the calculation of vaccine efficacy (VE) a

Vaccination status

Clinical status Vaccinated Unvaccinated Unknown

Unwell (ill)

Well

a

d

b

e

~~b a

VE ARU-ARV (b+e) (a d)x 100ARU bxOO(b + e)

a See text for details.

(e.g., 9-35 months) may be selected from each cluster(10). The vaccination status and any history of anillness clinically compatible with measles are deter-mined for each participant; vaccination status isassessed preferably from vaccination records.

If the attack rates are high so that the number ofcases in the sample is large, vaccine efficacy can becalculated directly from the coverage survey as shownin Table 1. Vaccine efficacy (VE) can also beexpressed in the form of relative risk (RR), which isthe ratio of ARV to ARU, as shown below:

VE ARU - ARV x 100 ( ARV x 100V= ARU ARU,= (I - RR) x 100

If the number of cases is low, they can be sup-plemented by other cases found through disease sur-veillance systems. This will increase the precision ofthe estimate (i.e., decrease the width of the confidenceinterval). On the assumption that the additional casesobtained from the surveillance system are representa-tive of the cases from the population surveyed, thenthe relative risk may be calculated directly (see Table2) without calculating ARV and ARU. This is theequivalent of a case exposure study in chronic diseaseepidemiology (11).

Methods. The following should be taken intoconsideration.(1) Case definition: same as in previous section (seeabove).

(2) Case ascertainment: if the attack rates are high,the number of cases in the sample may be sufficient toaccurately estimate vaccine efficacy. If the attackrates are low, other surveillance data can be used togain additional cases and increase precision.

(3) Vaccination status determination: preferablyfrom written records of persons in the sample and allcases.

(4) Prior disease: same as in previous section (seeabove).

Analyses. Attack rates and vaccine efficacy can becalculated as in an outbreak investigation (see Table1) when the coverage survey above is used. If sup-plemental cases are added, the calculation shownin Table 2 is used.

Secondary attack rates in familiesThe possibility that the exposure of vaccinees and

non-vaccinees to the disease during outbreaks may bedifferent can result in biased estimates of vaccineefficacy and is a potential problem with suchinvestigations. An alternative approach to reduce thisbias is to measure the secondary attack rates of thedisease in family members of index cases. Studies onmeasles have demonstrated that secondary attackrates in non-vaccinees are generally consistent fromfamily to family, implying that within the householdthere is generally uniform exposure (12, 13).Secondary attack rate determinations have not beenused as frequently as outbreak investigations inmeasuring the efficacy of measles vaccine (6, 14).Nevertheless, the technique has been thoroughlyevaluated and has proved useful not only for measles,but also for other vaccine-controlled diseases such as

pertussis (15, 16). An additional advantage of the

Table 2. Calculation of vaccine efficacy (VE) using coverage survey data and supplemental information on casesa

Vaccination status

Clinical status Vaccinated Unvaccinated Unknown

Unwell (ill) from coverage survey a b cTotal in coverage survey d e fOther ill from population surveillance g h iTotal ill a+g b+h c+i

VE=(1-RR)x100= 1 - (b+h Xl00

a See text for details.

1059


secondary attack rate method is that vaccinees andnon-vaccinees from several families can be added todetermine the overall attack rates in the vaccinatedand unvaccinated populations, provided the samedefinitions for cases and immunization status areused.To minimize the effect of previous exposure to

measles disease, the age group studied should berestricted to between 9 and 35 months.


(1) Case definition: same as described above.

(2) Case ascertainment: as described above underoutbreak investigations, a good population basedsurveillance survey should be carried out. Otherwise,families with single cases may be less likely to bereported than families with multiple cases. Formeasles, all cases in a given family should be listed bythe date of onset of the rash. Families should befollowed up for at least 18 days after the onset of rashin the first case in the family, the maximum intervalfor secondary cases being determined from earlierstudies of measles (12). Persons should be classifiedas cases or non-cases by their status 18 days afteronset of the first case in the family. For other diseases,the appropriate maximum incubation period can besubstituted for the 18-day interval in the case ofmeasles.

(3) Vaccination status determination: same asdescribed above under outbreaks. For measles,persons should be classified as vaccinated orunvaccinated by their status on the day of onset of therash in the first case in the family. For other diseases,the cut-off should be determined after considering theaverage incubation period for the disease in questionand the average time required for the vaccination tobecome effective.(4) Prior disease: same as described above.

Analyses. These must be based on the following:(1) For measles, persons in the family who develop arash within the first six days following onset of rash inthe first (index) case are considered as co-primarycases. The first case in a family and all co-primarycases should be excluded from the analysis. A similarapproach should be taken with other diseases.(2) Length of the follow-up for all families should beat least 18 days (for measles) after the onset of rash inthe first case.

(3) All secondary cases in all families in the target agegroup (other than the index case and co-primaries) areadded to give the total number of cases, and all the

non-cases are added to give the total number withoutillness.

(4) Vaccine efficacy is calculated using the data inTable 1.

Secondary attack rate in clusters

A modified form of household investigation hasbeen used in urban and semi-urban settings. Thistechnique has been less well studied than outbreakinvestigations and secondary attack rates in families.eIt is less rigorous (because comparability of exposureof vaccinees and non-vaccinees is less definite) but islogistically easier to carry out than intra-householdstudies. In the course of an outbreak, or towards theend of the measles transmission season, the studymust be conducted in a group of neighbourhoodclusters in each of which at least one known case ofmeasles occurred during the most recent transmissionperiod or some other specified period. The studysubjects (e.g., children between 9 and 35 months) arethose who live in close proximity to a known case, i.e.,no more than one house away from the open area infront of the doorway of the house with a case.

Clusters are defined operationally by the investi-gators, who start at the household of an identifiedcase,f then proceed to the neighbouring householdslisting all children in the target age group. If anothercase which had occurred during the outbreak period isfound in one of the visited households, the house-holds adjacent to it are visited, and so forth, until nofurther cases are found. Thus all the children investi-gated will have an equally close neighbourhoodrelationship to a case. This approach requires asecond visit to the neighbourhood clusters at least 18days later for confirmation of cases seen too early todetermine whether they met the case definition and todetect any secondary cases among their contacts.

Methods. The following should be taken intoconsideration.(1) Case definition: same as described above.(2) Case ascertainment: all cases in the target agegroup found in the surveyed households during thepredetermined time period are included, as well as thecase which led to studying the cluster.

(3) Vaccination status determination: same as insecondary attack rates in families.

(4) Prior disease: same as described above.

e DONDERO, T. J. ET AL. Efficacy of measles vaccination inCameroon (manuscript in preparation).

f The index case may be reported from any source includinghospitals, clinics, schools or a population based surveillancesurvey.


Analyses. See above, as described under outbreakinvestigations.

Coverage survey methods in endemic areas

Vaccine efficacy can be assessed in the absence of adefinable outbreak in urban populations with highlyendemic measles by using coverage survey methods.Conceptually this approach is similar to that used inoutbreak investigations except that vaccination statusis ascertained as of a given age (e.g., 12 months)rather than as of the beginning of the outbreak, anddisease history is ascertained up to the current age ofthe children in the survey rather than over a shorteroutbreak period of time. No actual outbreak isrequired. However, because the interval from diseaseto the time of the survey may be long (up to 2 years), ahistory taken from parents rather than specificclinical information is used to identify and define thecases.

This coverage survey approach is attractive becausethe situation it seeks to deal with is one frequentlyencountered in urban measles control programmes,and because the sampling techniques used are wellrecognized and easily applied. However, there hasbeen only limited experience in the use of thisapproach. Modifications have been introduced tominimize the biases, but more changes may berequired as additional experience accumulates. Themnethod presented below is an alteration of the clustersample design frequently employed to estimatecoverage (10). However, other random sampledesigns can also be used.

Thirty neighbourhood clusters are randomlyselected using the sampling methods described forvaccination coverage surveys. Fourteen or more 2-year-old children are sampled from each neighbour-hood cluster, and two new questions on previousmeasles disease and the age when this happened (ifapplicable) are added to the usual coverage surveyquestions on current age and date of vaccination. Ageat vaccination must be calculated from these data.The 2-year-old children sampled from each clustermust include at least 7 never-vaccinated children andat least 7 children vaccinated between 9 and 11months of age (or within three months of therecommended age at vaccination, if older than 9months).9

This approach will yield an estimate of the efficacy

8 If an estimate of vaccination coverage is desired for 2-year-oldchildren, the first seven such children encountered in each clustershould be used, regardless of their vaccination status or age atvaccination. Some of these children can be included in the group offourteen children being sought for a vaccine efficacy if theirvaccination status or age at vaccination qualify them for inclusion. Inthis manner, estimates of vaccine coverage and vaccine efficacy canbe obtained simultaneously using the familiar coverage surveysampling methods.

of vaccine when administered to children between 9and 11 months of age. To obtain an estimate ofmeasles vaccine efficacy when any interferinginfluence of maternal antibodies is low, 7 childrenvaccinated between 12 and 14 months of age couldalso be included in each cluster. This techniquerequires an expansion of the usual age groupemployed in coverage surveys from 12-23 months to24-35 months. If the usual coverage assessments aredesired in the 12-23-month age group, each clustershould include seven children in that age group aswell.With the information on the age when the disease

occurred among vaccinated and unvaccinatedchildren, it is possible to calculate the measles attackrates experienced by both groups of children from 12months of ageh up to their current age at the time ofthe survey, and the resulting vaccine efficacy.The attack rates are calculated as follows:

No. of cases after age 11 months'ARU = among never vaccinated children

Total number of never vaccinated children

No. of cases after age 11 monthsARV = among vaccinated children

Total number of vaccinated children

(ARU-ARV)VE (%) = ARU x 100


(1) Case definition. This approach assumes thatmeasles is sufficiently distinctive to be recognized asmeasles in areas of high endemicity by the mothers ofthe children surveyed. Since the recall period is twoyears or less, this is considered sufficiently recent forthe mother to recall the disease accurately. Any illnesswith a rash that was diagnosed as measles by themother is accepted as a case of measles. In Abidjan;Ivory Coast, where this method was developed, suchhistories have been shown to be reliable (17).

(2) Case ascertainment. All cases of measles reportedby questioning the mother are of interest, the deter-mination of the age (in months) at the time of theillness in those with positive histories requiringcareful attention. If the exact age cannot be elicited, adetermination of whether or not the disease occurred

h If children vaccinated between 12 and 14 months are included,their disease experience would be calculated starting as of 15 ratherthan 12 months.

i If the age of the vaccinated group changes, this should be ad-justed (see text).


before 12 months of age (or before 15 months if12-14-month-old vaccinees are also sampled) is theminimum information required. All cases whichoccurred among these 2-year-old children when theywere between 12 months and 2 years are included inthe study.(3) Vaccination status determination. Two-year-oldchildren vaccinated between 9 and 11 months of ageconstitute the principal vaccinated group included inthe survey. Children vaccinated prior to this age areexcluded from the analysis. Children vaccinated afterthis age should also be excluded from the analysisunless they are a specific target group included in thesurvey for vaccine efficacy purposes.(4) Prior disease. Some children vaccinated at 9-11months will have a history of measles prior to 12months of age either before or after their vaccination.Some unvaccinated children will also have a history ofmeasles prior to 12 months. All vaccinated andunvaccinated children with such histories should beexcluded from the numerators, but not from thedenominators of the appropriate rates calculated.Such a procedure will ensure that a minimally biasedestimate is obtained. It assumes only that childrenwith a history of measles disease are as likely to obtainvaccination as children without such a history.

Analyses. These must be based on the following:

(1) Persons with uncertain vaccination or diseasehistories are excluded.(2) Data obtained by using the criteria described inthe above (Methods) section are used to calculatevaccine efficacy.

Case-control studiesCase-control studies can be most useful when

personal immunization records are not generallyavailable but some other source such as records fromone or more clinics can be obtained. Intensive effortcan be made to determine the vaccination status of alimited number of cases and non-cases (controls)instead of concentrating on the whole population atrisk.The traditional vaccine efficacy equation cannot be

used in such studies. Cases in a case-control studyrepresent one sampling fraction of all cases and thecontrols represent a different sampling fraction of thepopulation that is not ill (18). In general, thesesampling fractions are unknown so that the totalpopulations of vaccinated and unvaccinated personscannot be calculated, thus preventing calculation ofthe attack rates.The vaccine efficacy equation can be expressed in

the form of relative risk (RR). In case-control studies

the RR can be approximated by the odds ratioi andvaccine efficacy can be calculated. By knowing thevaccination histories of cases and of non-cases(controls), the odds ratio and vaccine efficacy can beestimated. Case-control studies have not beenthoroughly evaluated in the measurement of vaccineefficacy. The results of some studies of measles andrubella outbreaks suggest that they reflect the vaccineefficacy accurately; with further use, refinementsmay be made (19).k In addition, case-control sets canbe added together from several outbreaks to increasethe numbers and the power of the calculations.

Methods. The following should be taken intoconsideration.(1) Case definition: same as described in outbreakinvestigations.

(2) Case ascertainment: same as described inoutbreak investigations, although all cases need notbe detected.(3) Vaccination status determination: this is onlynecessary for selected cases and selected controls(non-cases); otherwise, the same as described inoutbreak investigations.(4) Control selection. Matched pair analysis: onecontrol for every case should be selected and matchedwith the case for age, sex and residence. The controlsshould be well at the time of the investigation andshould preferably be selected at random fromsurrounding houses in the village. This offers theconvenience of choosing the controls on the sameoccasion as the cases are interviewed. Potentialcontrols should continue to be identified until one isfound which meets the matching criteria. Formeasles, the cases should be aged 9-35 months andthe controls in the same range as well as within 2months of the corresponding cases. Close matching inages is most important for cases .18 months old.Records of each case-control pair should be kept.Unmatched analysis: controls are selected ran-

domly from the affected village or villages in the agerange 9-35 months. The total number of controlsshould be one to two times the number of cases. Theunmatched approach may be more difficult thanmatched sampling since it requires the planning and

i The odds ratio is the ratio of the odds that a case is vaccinateddivided by the odds that a control is vaccinated. See Table 3 forcalculation of the odds ratio; this ratio may not approximate therelative risk when attack rates are high. When the attack rates in thevaccinated are greater than 100o, the vaccine efficacy will beerroneously high. In most instances the attack rates in the vaccinatedwill be < 1007o so that this error will not be important.

k ORENSTEIN, W. A. ET AL. Vaccine efficacy: a new applicationof case-control and case exposure methodology. Paper presented atthe Society for Epidemiologic Research meeting, Cincinnati, Ohio,16-18 June 1982.

1063FIELD EVALUATION OF VACCINE EFFICACY

Table 3. Determination of vaccine efficacy (VE) in acase-control study by (A) matched pair analysis and (B)unmatched analysis

Controls

Vaccinated Unvaccinated

A. Matched pair analysisVaccinated cases j pUnvaccinated cases k q

RR = odds ratio =k

VE (%)=(1 -RR)x 100=

(1 - x 100

Cases Controls

B. Unmatched analysisVaccinated a b

Unvaccinated c d

RR _odds ratio = adbc

VE (%)=(1 -RR)x 100=

(bad) 100

execution of a separate programme to randomly pickand interview the controls.

(5) Prior disease: cases and controls with histories ofprior disease should not be excluded from thecalculation (20).

Analyses. Matched pairs: (i) Case and controlpairs, in which the vaccination status of either one isunknown, should be analysed separately. (ii) Table3A shows an analysis of the matched case-controlpairs. Instead of individuals, each cell contains pairs.For example, j represents the number of pairs inwhich both the case and control were vaccinated whilep represents the number of pairs where the controlwas unvaccinated and the case was vaccinated. Thesum of j+k+p+q is equal to half the number ofparticipants. The odds ratio equals p divided by k.

Unmatched design: Table 3B shows how tocalculate vaccine efficacy from unmatched data.

Confidence intervals for vaccine efficacy

Confidence intervals for vaccine efficacydeterminations are shown in Table 4 (21). Theformulas in Table 4A apply to outbreak investi-gations and secondary attack rates in households or

clusters. Approximate confidence intervals can alsobe obtained for efficacy using cluster samples.Formulas for approximate confidence intervals

using supplementary cases and cluster samples inoutbreaks can be obtained from Hogue et al.(1J) (seeunder "case exposure studies"). The same referencecan be used for confidence intervals for case-controlstudies supplemented by Bayes' theorem.

Confidence intervals for case-control studies usingmatched pair analysis are shown in Table 4B (22).

DISCUSSION

The efficacy of vaccines in clinical use can be deter-mined by a variety of means including screening, out-break investigations, secondary attack rates infamilies or clusters, vaccine coverage assessments,and case-control studies. They all offer a means ofmonitoring vaccine programmes under conditions ofday-to-day vaccine use.The different techniques for measuring efficacy are

summarized in Table 5. The screening technique is themost useful and rapid means of determining whetherthere is a problem with a vaccine. All that is needed isa reliable estimate of the proportion of casesoccurring in vaccinated individuals and an estimate ofthe vaccine coverage in the population at risk. If theestimated efficacy is within expected limits, moredetailed studies are not warranted. However, if theresults suggest low efficacy, more rigorous methodsare needed to assess the efficacy more accurately.Of the more accurate methods available, outbreak

investigation offers the simplest means of measuringvaccine efficacy and is the preferred technique if thesituation permits. The biases inherent in the methodcan be minimized, particularly if the disease incidencerate is high during the outbreak and accurate recordsexist. A low measles incidence rate prior to the out-break is important, so the age group chosen should benarrow (e.g., 9-35 months) and rural areas wheremeasles is less likely to be endemic are best used. Inlarge populations, the underlying immunizationstatus prior to the outbreak can be estimated using thesame cluster sampling method used in coverageassessments.

Calculation of secondary attack rates in families isalso an excellent and accurate means of measuringvaccine efficacy and is an acceptable alternative to theoutbreak investigation. Secondary attack rates inclusters are also probably useful although furtherevaluation is needed.

Vaccine coverage methods in endemic areas arebest suited to urban areas where the measles incidencerate is high after age 11 months and low before 12months, and maternal histories of disease are thought


Table 4. Determination of 95% confidence intervals for vaccine efficacy estimates

A. All studies except case-control and outbreak investigations using cluster samples and supplementary cases.

Vaccinated

Unvaccinated

Cases Non-cases Total

a b a+b

c d c+d

Total a+c b+d a+b+c+d

RR- a / c(R + b) /(c + d)

VE (%) = (1 -RR)x 100(where RR = relative risk)

To get 95% confidence intervals the following formulas are used:

(1) for lower limit of VE: ( 1- a ) +1 (cd)

upper limit of relative risk=RRu=(RR) exp +1.96i a+b +__

+d )

lower limit of VE = (1- RR u) x 100 \

(2) for upper limit of VE: ( 1- ( a ) 1 c \

lowerlimitofrelativerisk=RRL=(RR)exp -1.96 a+b ) + c+d / j

upper limit of VE =(1 - RR L) X 100

B. Case-control studies: matched-pair analysis.Controls

Vaccinated Unvaccinated

Vaccinated cases j p

Unvaccinated cases k q

Relative risk (RR)_ odds ratio =p/k

The upper limit of RR = RRu= 1-Pu

and the lower limit of VE = (1 - RR u) x 100

where Pu b= 42-a and2a

a = (p + k) x (p + k + 3.84)b= (p+k) x (2 (p+ 1 )+3.84)C = (p + 1)2

The lower limit of RR = RR L= PLl-_PL

and the upper limit of VE = (1 - RR L) x 100

d- J/d2-~4aewhere PL = 2a and2a

a = (p + k) x (p + k + 3.84)d= (p+k) x (2 (p- 1 )+ 3.84)e=(p- 1)2

1065FIELD EVALUATION OF VACCINE EFFICACY

Table 5. Summary of techniques for measuring vaccine efficacy with their main advantages and disadvantages

Technique Comments Advantages Disadvantages

1. For measles, if the estimate is 1. Rapidwithin expected levels, no furtherinvestigations are needed.

2. If estimate < expected levels, 2. Requires few resourcesmore accurate techniquesare needed

1. Estimates may be inaccurate ifthe proportions of populationvaccinated and casesvaccinated are inaccurate

2. Outbreakinvestigations(a) Total census

(b) Clustersamples

1. The preferred technique,situation permitting

2. Indicated during outbreaks insmall populations whereimmunization and disease statusof all individuals can be assessed

3. Biases can be kept to a minimumusing a population-basedsurveillance survey in an areawith high attack rates and a lowincidence of disease before theoutbreak

1. Mostly indicated during largeoutbreaks in large populationswhen determination ofvaccination and disease statuson all individuals is not feasible.Otherwise same as 2 (a)

1. Most frequentlyevaluated technique

2. Allows collection of clinicalinformation on cases formore accurate diagnosis

1. Requires substantially moreresources than screening

2. Exposure of vaccinees and non-vaccinees to disease may not beabsolutely equivalent

3. With high attack rates, theexposure of vaccinees andnon-vaccinees to diseasebecomes more comparable

4. One of the easiest of the moreaccurate methods to perform

1. Same as for 2 (a) 1. Same as for 2 (a)

2. Because samples are taken ratherthan a census, there may besome loss in precision of theestimate

1. Next to outbreak investigations, 1. Corrects for potentialthis technique has been evaluated differences in exposuremost and is an acceptable between vaccinees andalternative non-vaccinees

2. Potentially one can addresults of many differentfamily investigations indifferent areas together,allowing more accurateestimates

1. Probably requires more resourcesthan for outbreak investigations,since each family must befollowed for at least 18 days,e.g., often >2 visits

2. Only a small number of childrenwill be in the right age group in agiven family so that many familiesmust be visited

3. Case definition will be lesspredictive for measles in theabsence of an outbreak

4. Secondary attack 1. Modification of the secondaryrates in clusters attack rates in families using

localized neighbourhoods

2. Most indicated when resourcesfor family investigations arelimited. Because the numbers ofpersons exposed in a cluster aregreater than in a family, fewervisits are needed

1. May correct for potentialdifferences in exposurebetween vaccinees and non-vaccinees in outbreakinvestigations.

2. Assuming the exposurewithin clusters is comparablefrom one cluster to another,the results of multipleclusters can be addedtogether

1. Same as for technique 3

2. Needs further evaluation ofuniformity of exposure withinclusters

Table 5: continued on next page

1. Screening

3. Secondary attackrates in families


Table 5: continued

Technique Comments Advantages Disadvantages

5. Vaccine efficacy 1. Modification of routine coverage 1. Requires only minor changes 1. Relies on the parent's diagnosisusing coverage assessment using older age of a technique with which and recall of the disease rathermethods in groups, 24-35 months, and most EPI personnel are familiar than clinical informationendemic areas adding questions on disease

history

2. Most indicated in non-outbreak 2. Requires similar resources as 2. If coverage assessments insettings when the disease is in coverage survey 1 2-23-month-olds are desired,highly endemic, such as in large the number of children per clusterurban areas might have to be increased

3. With minor changes indesign it will provide coveragedata simultaneously

6. Case-control 1. The vaccination histories of 1. Allows maximal resources to 1. Will give a falsely high vaccinestudies cases and matched non-cases be utilized in finding the efficacy if the attack rates in the

are compared. Vaccine efficacy vaccination status of cases vaccinated are highis calculated by using the and a few matched controls,odds ratio to approximate the instead of the entirerelative risk population

2. Most indicated when vaccinationstatus is difficult to obtain, aswhen individual immunizationrecords are poor, but anothersource such as one or moreclinics have records

to be accurate. This technique has not been usedwidely and refinements may be made as a result ofgreater experience.

Case-control studies are best suited to areas wherereliable personal immunization records may be diffi-cult to find but other information, such as clinicrecords, may be available. In this way, intensiveefforts can be applied to determining the vaccinationstatus of the cases and a few selected controls insteadof the entire population at risk.

It may be noted that no epidemiological method isperfect because it cannot exactly duplicate the experi-mental conditions of a prospective randomizedclinical trial. The most accurate results will beobtained when biases are anticipated and correctivemeasures are taken whenever possible.

Clinical vaccine efficacy determinations are carriedout in order to assess whether the observed pattern ofillness is consistent with the proper use of a highlyeffective vaccine. The results can also be used to makechanges in the programme if necessary. A lower thanexpected efficacy should lead to a careful evaluationof the vaccine management and vaccine adminis-tration technique. If these are unsatisfactory, correc-tive measures should be taken. If satisfactory, otherexplanations should be sought, e.g., a transientproblem due to a fault in a single lot of vaccine or asingle shipment.The components of a vaccine efficacy evaluation-

case definition, case ascertainment, and vaccinationstatus determination -apply to studies on allvaccines. Case definitions will vary depending on thedisease and some will require more laboratorysupport than others (23).' Similarly, caseascertainment should generally be population basedrather than clinic or hospital based. Diseases withhigh proportions of infections that are subclinical,such as poliomyelitis, can be evaluated solely on theclinical illness rather than total infections. If theproportion of subclinical infections is assumed to bethe same in the unvaccinated group and amongvaccine failures, vaccine efficacy will be accuratelyreflected by the number of clinical illnesses alone.The general methodology can be applied to

vaccines requiring multiple as well as single doses.The efficacy of each dose can be calculated using theattack rate in the unvaccinated (ARU) with no priordoses, and compared in successive calculations to theattack rates in recipients of 1 prior dose, 2 prior doses,3 prior doses, etc. Care must be taken to ensure thatboth numerator and denominator reflect the samegroup.Each component of the vaccine efficacy evaluation

is potentially associated with problems that can leadto substantial biases in the estimate of vaccineefficacy. Awareness of these potential biases can lead

' See footnote b on p. 1056.


to corrective measures to keep them to a minimum. Ingeneral, if a method cannot be totally corrected, thetechniques recommended will tend to slightly under-estimate the vaccine efficacy. On occasion, this maylead to intensive investigations of vaccine handlingpractices and other aspects of the immunizationprogram which may not really be necessary.However, it is better occasionally to investigateunnecessarily than to fail to investigate when an

intensive examination may be important.Clinical vaccine efficacy studies provide useful

information to health care providers concerning theeffectiveness of vaccines and can help in theevaluation of policy decisions and the determinationof trouble spots in vaccine programmes. In addition,the coordinators of routine immunization pro-grammes will be able to increase the public'sconfidence in immunization.

RtSUMt

EVALUATION SUR LE TERRAIN DE L'EFFICACITE DES VACCINS

II est possible d'evaluer l'efficacite des vaccinationspratiqu6es sur le terrain en mesurant par des moyens epi-demiologiques le degre de protection contre la maladie. Cestechniques sont particulierement utiles lorsque l'efficacit6de la vaccination est mise en doute. Ce probleme gagne enacuite a mesure que la couverture vaccinale s'etend, car unnombre de plus en plus grand de cas s'observera chez despersonnes vaccin6es, meme lorsque l'efficacite du vaccin estbonne. On peut obtenir une estimation rapide de l'efficaciteen utilisant les donn&es existantes sur la proportion de cassurvenant chez des sujets vaccines et l'estimation actuelle dela couverture vaccinale dans la population a risque. Si cettepremiere approche montre une faible efficacite, desmethodes utilisant la collecte de nouvelles donn&es etdonnant une estimation plus precise sont indiqu&es. Cesmethodes s'appuient sur les enquetes sur les flamb6es, even-tuellement avec echantillonnage en grappe, les tauxd'atteinte secondaire dans les familles, les taux d'atteintesecondaire dans les grappes, les enquetes sur la couverture

vaccinale dans les zones d'end6mie, et les etudes r6trospec-tives. Quand les circonstances le permettent, les enquetes surles flamb6es constituent la m6thode de choix, notammentpour 1'6valuation de l'efficacit6 du vaccin antirougeoleux.Les taux d'atteinte secondaire dans les familles constituentaussi une excellente methode. Les autres methodes pre-sentent des avantages dans certains cas. Quelle que soit lam6thode choisie, les estimations les plus precises de l'effica-cite des vaccins s'obtiennent lorsque les biais potentiels sontpris en compte et reduits au minimum lors de la conceptionde l'enquete ou de l'6tude. Les r6sultats qui se situent dansles limites prevues doivent susciter une conflance accruedans le programme de vaccination. Une efficacite plus faibleque prevu au contraire doit amener a entreprendre desenquetes pouss6es sur les pratiques en matiere demanipulation des vaccins et sur d'autres aspects du pro-gramme de vaccination pouvant etre A l'origine de la perted'efficacite.

REFERENCES

1. MARKS, J. S. ET AL. Methodologic issues in the evalu-ation of vaccine effectiveness. Measles vaccine at 12versus 15 months. Am. j. epidemiol., 166: 510-523(1982).

2. HALSEY, N. A. The optimal age for administeringmeasles vaccine in developing countries. In: Halsey,N. A. & de Quadros, C., ed., Recent advances in immu-nization, a bibliographic review (Scientific PublicationNo. 451), Washington, Pan American Health Organ-ization, 1983, pp. 4-17.

3. EXPANDED PROGRAMME ON IMMUNIZATION. The optimalage for measles immunization. Wkly epid. rec., 57:89-91 (1982).

4. CENTERS FOR DISEASE CONTROL. Measles vaccineefficacy-United States. Morb. Mortal. Wkly Rep., 29:470-472 (1980).

5. LINNEMANN, C. C. JR. Measles vaccine: immunity,reinfection and revaccination. Am. j. epidemiol., 97:365-371 (1973).

6. MCCORMICK, J. B. ET AL. Measles vaccine efficacydetermined from secondary attack rates during a severeepidemic. J. pediatr., 90: 13-16 (1977).

7. MARKS, J. S. ET AL. Measles vaccine efficacy in childrenpreviously vaccinated at 12 months of age. Pediatrics,62: 955-960 (1978).

8. HULL, H. F. ET AL. Measles mortality and vaccineefficacy in rural West Africa. Lancet, 1: 972-975(1983).

9. PETO, R. ET AL. Design and analysis of randomizedclinical trials requiring prolonged observation of eachpatient. II. Analysis and examples. Br. j. cancer, 35:1-39 (1977).

10. HENDERSON, R. H. & SUNDARESAN, T. Cluster samplingto assess immunization coverage: a review of experiencewith a simplified sampling method. Bull. Wld HlthOrg., 60: 253-260 (1982).


11. HOGUE, C. J. R. ET AL. Estimators of relative risk forcase-control studies. Am. j. epidemiol., 118: 396-407(1983).

12. HOPE-SIMPSON, R. E. Infectiousness of communicablediseases in the household (measles, chickenpox, andmumps). Lancet, 2: 549-554 (1952).

13. Top, F. H. Measles in Detroit, 1935. 1. Factorsinfluencing the secondary attack rate amongsusceptibles at risk. Am. j. public hfth, 28: 935-943(1938).

14. MCINTYRE, R. C. ET AL. Measles and measles vaccineefficacy in a remote island population. Bull. Wld HlthOrg., 60: 767-775 (1982).

15. MEDICAL RESEARCH COUNCIL. The prevention ofwhooping cough by vaccination. Br. med. j., 1: 1463-1471 (1951).

16. BROOME, C. V. ET AL. Epidemiology of pertussis,Atlanta, 1977. J. pediatr., 98: 362-367 (1981).

17. BERNIER, R. H. ET AL. A new practical approach todetermining measles vaccine efficacy. Am. j.epidemiol., 118: 410-411 (1983).

18. SCHLESSELMAN, J. J. Case-control studies. Design,conduct, analysis. New York, Oxford University Press,1982, pp. 36-38.

19. GREAVES, W. L. ET AL. Clinical efficacy of rubellavaccine. Pediatr. infectious disease, 2: 284-286 (1983).

20. SMITH, P. G. ET AL. Assessment of the protectiveefficacy of vaccines against common diseases usingcase-control and cohort studies. Int. j. epidemiol., 13:87-93 (1984).

21. KLEINBAUM, D. G. ET AL. Epidemiologic research.Principles and quantitative methods. Belmont, CA,Lifetime Learning Publications, 1982.

22. SCHLESSELMAN, J. J. Case-control studies. Design,conduct, analysis. New York, Oxford University Press,1982, pp. 209-211.

23. GEMERT, W. ET AL. Agents affecting health of motherand child in a rural area of Kenya. The diagnosis ofmeasles under field conditions. Trop. geogr. med., 29:303-313 (1977).

Documents

Field evaluation of - MMV