Kaplan, 1958. Nonparametric Estimation From Incomplete Observations

8/13/2019 Kaplan, 1958. Nonparametric Estimation From Incomplete Observations.

1/26


2/26

NONPARAMETRIC ESTIMATION FROMINCOMPLETE OBSERVATIONS*

E. L. KAPLANUniversity of California Radiation Laboratory

ANDPAUL MEIER

University of ChicagoIn lifetesting, medical follow-up, and other fields th e observation of

the t ime of occurrence of the event of interest called a death may beprevented for some of the items of th e sample by the previous occurrence of some other event called a loss . Losses may be either accidentalor controlled, th e latter resulting from a decision to terminate certainobservations. In either case it is usually assumed in this paper that thelifetime age at death is independent of the potential loss time; inpractice this assumption deserves careful scrutiny. Despite the resultingincompleteness of the data, it is desired to estimate the proportion petof items in the population whose lifetimes would exceed t in the absenceof such losses), without making any assumption about the form of thefunction pet). The observation for each item of a suitable initial event,marking the beginning of it s lifetime, is presupposed.

For random samples of size the product-limit PL estimate canbe defined as follows: List and label the observed lifetimes whetherto death or loss) in order of increasing magnitude, so that one haso::;t, ::;t2 :: ; ::;tN . Then pet) IIr [ N -r / N -r + 1 )] where rassumes those values for which tr ::;t and for which tr measures thetime to death. This estimate is th e distribution, unrestricted as to form,which maximizes the likelihood of the observations.

Other estimates that are discussed are th e actuarial estimates whichare also products, but with the number of factors usually reduced bygrouping ; and reduced-sample RS) est imates , which require thatlosses not be accidental, so that the limits of observation potential losstimes) are known even for those items whose deaths are observed. Whenno losses occur at ages less than t, th e estimate of pet) in all cases reduces to the usual binomial estimate, namely, the observed proportionof survivors.

CONTENTS1. I n t r o d u c t i o n 4581.1 o rmu l a ti o n 4581.2 Nonparametric estimation 4591.3 Examples of th e RS and PL estimates 4591.4 No t a t i o n 4612. The Product-Limit Estimate 462

2.1 Definition and calculation 4622.2 Mean and variance of pet 4652.3 Mean lifetime 4673. The Reduced-Sample Estimate vs. the PL , 4693.1 Alternatives to the PL estimate 4693.2 Dependence of deaths and losses 470Prepared while th e authors were at Bell Telephone Laboratories an d Johns Hopkins University respectively.

The work was aided by a grant from th e Office of Naval Research.457

Downl

oadedby[PurdueUniversity]at03:3520March2013


3/26

458 AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 19584. Actuarial Estimates.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4714.1 Estimates using only n il A 4714.2 Estimates using average ages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4735. Maximum Likelihood Derivation of the PL 4756. Means and Variances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4766.1 The PL estimate P t 4766.2 Covariances and mean lifetimes 4787. Consistency; Testing with Replacement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479

1. INTRODUCTION1.1 Formulation. In many estimation problems it is inconvenient or impossible to make complete measurements on all members of a random sample.

For example, in medical follow-up studies to determine the distribution of survival times after an operation, contact with some individuals will be lost beforetheir death, and others will die from causes it is desired to exclude from consideration. Similarly, observation of the life of a vacuum tube may be ended bybreakage of the tube, or a need to use the tes t facilities for other purposes. Inboth examples, incomplete observations may also result from a need to get outa report within a reasonable time.

The type of estimate studied here can be briefly indicated as follows. Whena random sample of N values, T I T2 , TN of a random variable is given,the sample distribution function F t) is naturally defined as that which assignsa probability of l to each of the given values, so that F t equals l timesthe number of sample values less than the argument t. Besides describing thesample, this F t is also a nonparametric estimate of the population distribution,in the sense indicated in 1.2 below. When the observations are incomplete, thecorresponding estimate is still a step-function with discontinuities at the agesof observed deaths, but it can no longer be obtained as a mere description ofthe sample.

The samples considered in this paper are incomplete in the sense that onehas given, not a random sample T I , TN of values of the random variableT itself (called the lifetime), but the observed lifetimes

t = min Ti , L i ) , i = 1,2, N. laHere the Iu, called limits of observation are constants or values of other randomvariables, which are assumed to be independent of the T i unless otherwisestated (in Sections 3.2 and 7). For each item it is known whether one has

ort.. < r;

t i = I , (a death

t = L, (a loss .

lb

Ordinarily the T i and L are so defined as to be necessarily nonnegative.The items in the sample are thus divided into two mutually exclusive classes,namely deaths and losses. A loss by definition always precludes the desiredknowledge of T i On the other hand, a death does not always preclude theknowledge of the corresponding L i , in case the limits of observation are non

random and foreseeable. Such knowledge of the L may have value; for example,

Downl



4/26

NONPARAMEJTlUC EJSTIMATION 9it makes available the reduced-sample estimate Section 3 , if one chooses touse it.The type of sample described is a generalization of the censored sample defined by Hald [17J, and a specialization of the situation considered by Harris,

Meier, and Tukey [18J.The term death has been adopted as being at least metaphorically appropriate in many applications, but it can represent any event susceptible of randomsampling. nparticular, the roles of death and random loss may be interchangeable. By redefining the classification of events into deaths and losses, itmay be possible to approach the same data from various points of view and thusto estimate the survivorship functions pet that would be appropriate to various categories of events in the absence of the others. This is familiar enough;see for example [6], [13], [16J.

1.2 N onparametric estimation. ostgeneral methods of estimation, such asmaximum likelihood or minimum chi-square, may be interpreted as proceduresfor selecting from an admissible class of distributions one which, in a specifiedsense, best fits the observations. To estimate a characteristic or parameterof the true distribution one uses the value that the characteristic has for thisbest fitting distribution function. seems reasonable to call an estimation procedure nonparametric when the class of admissible distributions from which thebest-fitting one is to be chosen is the class of all distributions. Wolfowitz [28]has used the term similarly in connection with the likelihood ratio in hypothesistesting . With a complete sample, it is easy to see that the sample distributionreferred to in 1.1 is the nonparametric estimate on the maximum likelihoodcriterion. The same result is true of the product-limit estimate for incompletesamples, as will be demonstrated in Section 5.The most frequently used methods of parametric estimation for distributionsof lifetimes are perhaps the fitting of a normal distribution to the observationsor their logarithms by calculating the mean and variance, and fitting an exponential distribution e: p tj fJ. by estimating the mean life j alone. Such assumptions about the form of the distribution are naturally advantageous insofar

as they are correct; the estimates are simple and relatively efficient, and acomplete distribution is obtained even though the observations may be restricted in range. However, nonparametric estimates have the important functions of suggesting or confirming such assumptions, and of supplying the estimate itself in case suitable parametric assumptions are not known. An important property of these nonparametric estimates is that if the age scale is transformed from t to t = j t) , where j is a strictly increasing function, then the corresponding estimated distribution functions are simply related by F* f t=F t).

1.3 Examples oj the RS and PL estimates. We will consider the followingsituation. A random sample of 100 items is put on test at the beginning of 1955;during the year 70 items die and 30 survive. At the end of the year, a largersample is available and 1000 additional items are put on test. During 1956,15 items from the first sample and 750 from the second die, leaving 15 and 250survivors respectively. Asof the end of 1956, it is desired to estimate the proportion P 2 of items in the population surviving for two years or more.

Downl



5/26

460 AMERICAN STATISTICAL ASSOCIATION JOURNAL JUNE 1958The survival probabilities are supposed to depend on the age the durationof the test rather than on the calendar year, and hence the data are arrangedas in Table 460.

TABLE 460Samples

Initial numbersDeaths in first year of ageOne-year survivorsDeaths in second year of ageTwo-year survivors

I

10070301515

1000750250

This particular example is such that it is easy to form an estimate P* 2= 15/100= 0.15 from the first sample alone. This is called the reduced s mpleRS estimate because it ignores the 1000 items tested only during 1956. is alegitimate estimate only when the reduced sample is itself a random sample;this will be the case only when as assumed here the observation limits twoyears for the first sample, and one year for the second are known for all items,deaths as well as losses. In the absence of this information, one would have nobasis for discriminating among the 835 deaths observed before the age of twoyears. One cannot simply ignore the 250 losses at age one year; since only 15items have survived for two years, P 2 would then be estimated as 15/850 .018, an absurd result. The point is discussed further in 3.1 below.We now inquire whether the second sample, under test for only one year, canthrow any light on the estimate of P 2 . Clearly it will be necessary to assumethat both samples have been drawn from the same population, an assumptionthat the RS estimate P* 2 avoided. At any rate, the estimates of P l fromthe two samples, namely 0.30 and 0.25, are not sufficiently different to contradict the assumption. By combining the two samples, the estimate

F l = P* l 30 250 / 100 + 1000 .255is obtained for P l . In this case the RS has the same value as the other estimate to be discussed, the product-limit or PL. This result exhausts the usefulness of the second sample for the present purposes; how does it help to estimate P 2 ?The answer is that there are advantages to using the first the smallersample for estimating P 2 /P 1 , the conditional probability of survival fortwo years given survival for one year, rather than P 2 itself. This estimate is

F 2 /F l = 15/30 = 0.50, whenceF 2 = 0.255 X 0.50 = 0.127,a very simple example of the product limit PL estimate. The outstandingadvantage of this strategy is that it works just as well if we are not privilegedto know that the 750 deaths in the second sample had observation limits of oneyear, because these items are irrelevant to the estimation of P 2 /P 1 in any

Downl



6/26

NONPARAMETRIC ESTIMATION 461case. Other considerations for deciding between the two estimates will be setforth in Section 3.

The discussion of the PL estimate will be continued shortly, in Sections 2and 3. Section 3 is equally concerned with the RS estimate, while Section 4 isdevoted to the actuarial estimates. The remaining three sections consist principally of mathematical derivations. Though much older, the actuarial estimatesare essentially approximations to the PL they are products also, but typicallythey aim to reduce the number of factors by grouping. Grouping mayor maynot be possible for the PL itself. The distinguishing designation product-limitwas adopted because this estimate is a limiting case of the actuarial estimates.I t was proposed as early as 1912 by Bohmer [6] referred to by Seal [26] , butseems to have been lost sight of by later writers and not further investigated.1.4 Notation. The survival function

pet = Pr T t , cgiving the population probability of surviving beyond t, will be used in placeof the distribution function F t pet because of its convenience where theproduct-limit estimate and its actuarial approximations are concerned. Inaddition the following functions are defined:

Pet product-limit PL estimate of pet .P* t reduced-sample RS estimate of pet .net =the number of items observed and surviving at age t, when deaths

but not losses at t itself are subtracted off.N t = the expectation of net , for fixed observation limits.N t = the number of items having observation limits L such that L t. Inpractice this function is not necessarily known.For the first reading of the paper it may be desirable to suppose that the

death of one item and the loss of the same or any other item never occur at thesame age, and never coincide with an age t at which any of the above functionsare to be evaluated. This condition can always be met by fudging the ages alittle when necessary. On the other hand, a regular user of the techniques willprobably come to regard overt fudging as naive; he will prefer to formalize hisnotation and record-keeping by adopting the conventions already insinuatedinto the definitions of death and loss, pet , and net above. These conventionsmay be paraphrased by saying that deaths recorded as of an age t are treated asif they occurred slightly before t, and losses recorded as of an age t are treatedas occurring slightly after t. In this way the fudging is kept conceptual, systematic, and automatic.

The convention that deaths precede losses in case of ambiguity is based onthe following sequence of operations, which is clearly more efficient than thereverse sequence: Examine a group of items of age to observe the number Bofdeaths since the last examination, and then remove or lose contact with anumber X of the survivors. I t may then be convenient simply to record asthe age of death or loss of the B }.. items, especially if is always an integralmultiple of a fundamental time interval; in fact, however, the deaths will havepreceded the losses.

Downl



7/26

462 AMERICAN STATISTICAL ASSOCIATION JOURNAL JUNE 1958

Id

The chief exception to the immediate applicability of this convention occurswhen the losses are random but cannot affect items that have already died.Then the possible sequences of occurrence of deaths and losses between examination times assumed to be close together are approximately equally likely,and a reasonable compromise is to assume that half the losses in the intervalprecede and half follow the deaths, as in 4b below. the loss of an item iscompatible with the possibility of its having died unknown to the experimenter between the time when it was last examined and the time of loss, anitem lost in this way is effectively lost just after it was last examined, and theconvention is entirely appropriate. The disappearance of individuals subjectto medical follow-up is a case in point.

The remainder of the conventions concern the treatment of discontinuities inthe functions listed above. Should the value assigned to nCt say, for an argument t for which it is discontinuous be the right-hand limit n t+O = lim n t+has h-tO with h>O, the left-hand limit n t-O the same but with h


8/26

NONPARAMETRIC ESTIMATION 463 step (a) is left relatively arbitrary and approximations or parametricassumptions are accepted in step (b), one arrives at the actuarial estimates.

The PL is obtained by selecting the intervals in (a) so that the estimation in (b)is a simple binomial, without any recourse to assumptions of functional form.The condition for this is that within each interval, deaths and losses be segregated in a known fashion. As a beginning, it may be assumed that no intervalcontains both deaths and losses. Then if the number under observation justafter Uj-I is denoted by nj, and OJ deaths are observed in the interval Uj-I, Uj ,the estimate is clearly

i = nj - Gj)lnj = In;, (2a)where n; is the number under observation just after the OJ deaths. However, ifthe interval contains only losses (but at least one item survives throughout theinterval), the estimate is f = 1. the product of conditional probabilities formed in step (c), unit factors

may as well be suppressed; and we need not be concerned with the manner inwhich the losses are distributed among the intervals, so long as nj and n arecorrectly evaluated in (2a), and no losses occur at ages intermediate to the OJdeaths. The situation is illustrated by the following scheme:TABLE 463

No. of items N n,No. of deaths or losses > 0Division points Uo=n n nih >-. 02 >- 2

Here N is the initial number of items, and the braces join the numbers whoseratios are the conditional probabilities (2a). The numbers in the second lineare the differences of those in the first, the A scounting losses and the o s deaths;some of these could be zero. The division points Uj are placed in the third lineto show that the OJ deaths occur between Uj_1 and Uj, while Uj is located anywhere among the jlosses. The relation n; 5,n uj) 5,nj+1 holds.The PL estimate is now given by.P t = n; Inj) , with u = t, = n, - O 2b)

the greatest observed lifetime t* corresponds to a loss, (2b) should not beused with t>t*; in this case .P t can be regarded as lying between 0 and F t* ,but is not more closely defined.

it is desired to permit the entrance of items into the sample after the commencement of their lifetimes, this can be done by treating such entrances aslosses that are counted negatively in jThe same items can of course disappear again at a later age and so yield ordinary losses as well. t is assumed thatnothing is known of the existence of any such item that dies before it becomesavailable for observation; that is, the observation is censored on the right buttruncated on the left, in the terminology of Hald [17].

Downl



9/26

464 AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 1958The form 2b was selected for the estimate to give the minimum numberof elementary factors and the maximum grouping of the observations. Nevertheless, the number of deaths in an interval can easily be as small as unity.

The resulting estimate h though of limited value by itself, is none the lessacceptable as a component of P t . In fact, one is at liberty to take the intervalsas short and as numerous as one pleases, and to regard each death as occupyingan interval by itself. To specify the resulting expression, one relabels the N agesti of death or loss in order of increasing magnitude, and denotes them byt ~ t 2 ~ t N Then

P t = [ N r / N r+1 ], 2cwhere runs through those positive integers for which t/ t and t is the ageof death not loss . The cancellation of like integers in numerator and denominator where they occur reduces 2c to 2b . there are no losses, everythingcancels except the first denominator N and the last numerator n t , say, andthe reduces to the usual binomial estimate n t /N. 2c shows that P t is astep-function which changes its value only at the observed ages of death, whereit is discontinuous.In analyzing data on lifetimes by the multiplication of conditional probabilities, one of the following three procedures will usually suffice:1 the number of deaths is relatively small, these deaths may be arranged

in order of age without grouping, and the numbers of losses in the interveningage intervals counted. The estimate is calculated by 2c .2 1 is too time-consuming but the number of distinct ages of loss isrelatively small, these ages may be arranged in order, additional division pointsinserted as desired, and the numbers of deaths in the resulting age intervalscounted. some of these intervals are shorter than necessary and are found tocontain no deaths, they can be combined with adjacent intervals. The estimate is calculated by 2b .3 neither 1 nor 2 is compact enough, then division points are chosenwithout close consideration of the sample, deaths and losses are counted ineach interval, and an actuarial approximation to the PL, such as 4b , is used.As a miniature example of case 1 , suppose that out of a sample of 8 itemsthe following are observed:Deaths at 0.8,3.1,5.4,9.2 months.Losses at 1.0,2.7,7.0,12.1 months.The construction of the function P t then proceeds as follows:

TABLE 464Uj n j Ai P Ui0.8 8 7 2 7/83.1 5 4 0 7/105.4 4 3 1 21/409.2 2 1 0 21/80

12.1 1 1 1 21/80

Downl



10/26

NONPARAMETRIC ESTIMATION 465Each val ue of p Uj is obtained by multiplying n/ ln j by the preceding valuep Uj_l . The age 12.1 is recorded in the last line to show the point at whichp t becomes undefined; since it is a loss time, the 12.1 is enclosed in parentheses. is to be inferred from the table that 1 5.3) = 7/10, for example. Thethird and fourth columns could be omitted since n / =n j - l except in the lastline, which corresponds to a loss) and Aj=n/ -n jH.A rudimentary illustration of case 2) has already been given in 1.3. A lit tlemore el abor at e example of a similar sort with N = 100 is given in Table 465.Here 1.7, 3.6, and 5.0 are assumed to be the only ages at which losses occur;they are prescribed as division points. The other division poi nts 1, 2, 3, 4) areselected at pleasure, with the object of interpolating additional points on thecurve of p t vs. t. Nss is the effective sample size defined in 2j) below. Inpractice four columns headed Uj, nj, n/, Pi will suffice. From the table oneinfers that .74 t* as p t*) times thesurvival function for the supplementary sample, then the expected value of

Downl



11/26


12/26

NONPARAMETRIC ESTIMATION 467 t is often instructive to estimate an effective sample size LVE t), which in theabsence of losses would give the same variance V[1\t) Evidently

E t) = Pet) - p t ] jV[p t) ]. t can be shown that NE t) is nonincreasing as t increases, and that

net) / Pet) E t) D t) / - F t)],

2j)

2k)where D t) is the number of deaths observed at ages not exceeding i Theupper bound was proposed as an approximation to E t) by Cornfield [8J. Thelower bound corresponds to the reduced-sample variance.

In Table 464 of 2.1 we had P 6) =21 40=0 525 By 2f) the variance of P 6)is estimated asV[P 6)] = 0.525)2 ~ ~ = 0.042.8X7 5X3

By 2g) the same result appears in the form 1 1 1)0.525)2 .7X8 4X5 3X4

The effective sample size is estimated asi\TE 6) = 0.525) 0.475)/0.042 = 6.0.

The bounds for LVE 6) in 2k) are 5.7 and 6.3. Values of NE are also indicated inTable 465.2.3 Mean lifetime. The PL estimate j; of the mean lifetime ii is defined asthe mean of the PL estimate of the distribution. t is well-known and easilyproved by integrating by parts) that the mean of a nonnegative random variable is equal to the area under the corresponding survivorship function. Hence

J = i P t)dt.Of course, if Pet) is not everywhere determined, J is undefined. In cases wherethe probability of an indeterminate result is small, Pet) is practically unbiasedand the same is true of iiwe complete Pet) in the example of Table 464 by following the longestobserved individual with tr =12.1) to death at t=14.3 we have

Ii = 1.000) 0.8) 0.875) 3.1 - 0.8) 0.700) 5.4 - 3.1) 0.525) 9.2 - 5.4) 0.2625) 14.3 - 9.2)= 0.800 2.012 + 1.610 1.995 1.339 = 7.76.

et were completed by setting et =0 for the indeterminate range, thelast term in the above sum would be replaced by 0.2625) 12.1-9.2) =0.761,and ji would be estimated as 7.18.Of course, if the probability of an indeterminate result is high, there is nosatisfactory way to estimate JI In such cases Irwin [19J has suggested that in

Downl



13/26

468 AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 1958place of estimating the mean itself, one should estimate the mean life limitedto a time L, say ,u[Lj. This is the mean of min T i , L), with L chosen at the investigator s convenience. Naturally, one would choose L to make the probabilityof an indeterminate result quite small. one chooses to use this procedure heshould give an estimate of peL) along with } i [Lj we take L= 10 in ourexample,

,U[IOl = 0.800 2.012 1.610 1.995 0.2625) 10 - 9.2) = 6.63,and P 1O) = 0.2625.

In Section 6.2 an approximate formula is given for the variance of j i: f A2 t) IdP t I = f A2 t) IdP t I, 21)

,u 0 N t P t 0 NO t)P2 t)where A t = ft P u du. Upon making the obvious substitutions we find, aftersome reduction, the following estimate of V ,U):

A A r2V jl = N - r N - r + 1 2m)

where r runs over those integers for which i; corresponds to a death, andA r=it: P u du. there are no losses,

NA r = I ti - tr IN,and it can be shown that V ji) reduces to ti-T 2IN2. This fact, plus theimpossibility of estimating the variance on the basis of only one observeddeath, suggests that 2m) might be improved by multiplication by DI D- l ,where D is the number of deaths observed.

For our first estimate of Ii above we haveAl = 2.012 1.610 1.995 1.339 = 6.956,A 4 = 1.610 1.995 1.339 = 4.944,A o = 1.995 1.339 = 3.334A 7 = 1.339 = 1.339.

Obviously the A r and ji are best calculated in reverse order.) We then have forthe estimated variance,

A 6.956)2 4.944)2 3.334)2 1.339)2V m = = 3.917X 8 4X 5 X4 lXThe estimated standard deviation of.u is v3.91 = 1.98. the factor DI D - l=4 is included, these results become 5.21 and 2.28 respectively. one is limited to grouped data as in Table 465 it is necessary to useactuarial-type assumptions e.g., the t rapezoidal rule) to estimate Ii and itsvariance. Thus, wemay estimate the mean life limited to 5 time units as follows:

u[o] = 1/2 [ 1.00 0.97) 1.0 - 0.0) 0.97 0.92) 1.7 - 1.0) 0.52 0.27) 5.0 - 4.0)] = 3.76.

Downl



14/26

NONPARAMETRIC ESTIMATION 469We observe in conjunction with this estimate that the estimated proportionsurviving the l imit is F S =0.27.

3. THE REDUCED-SAMPLE ESTIMATE VS. THE PL3.1 Alternatives to the PL estimate. The acceptable nonparametric estimationprocedures known to the writers in the situation considered are variants of the

PL and of the reduced-sample RS) estimate defined byP* t = n t jN t = n t j[n t D t ]. 3a)

Here N t =n t +DO t is the number of items with observation limits ~ t ofthese, DO t die at ages ~ t and net survive beyond t. A simple example of theRS estimate has been given in 3 Berkson and Gage [3] have called it thead-hoc method. the lifetimes are independent of the observation limits, the reduced sampleof N t items will be a random sample and P* t a simple binomial estimate,unbiased, and with the variance

V [P* t ] = pet - pet ]jN t . 3b)

3c)

It has already been noted that this is equal to or larger than the approximation2h) to the PL variance. As with the PL, the estimatefJ = i P* t dtof the mean life may have to be truncated at the greatest of the observationlimits. In Section 6.2 it is shown that the variance of J.l* is given exactly byf 1 - pet f V fJ.* = 2 P u dudt. N t t

In practice one substitutes P* t for pet in this formula and perhaps approximates the integrals.

Whereas the PL estimate has discontinuities only at observed ages of death,it is apparent from 3a) that the RS generally has discontinuities at losses andobservation limits also. Furthermore, suppose that one of the observationlimits L is such that all or a sufficient number of) the corresponding items areobserved to die prior to L while other items survive and are observed beyond LThen the decrease in N t without an equivalent decrease in n t ~ will causeP* t to increase as t increases through L. Thus P* t , unlike Pet and the t rueP t , is not necessarily monotonic decreasing. However, this appears to be adisadvantage only in a psychological sense. It does not seem advisable to avoidit, as one could, by a further reduction of the sample, basing the estimatedpet) for all ~ suitable L on the fixed sample of items having observation

l i m i t s ~ LThe expression DO t =N t -net in 3a) is not the total number D t of

deaths observed prior to or at age i but the usually smaller) number of suchdeaths having observation l i m i t s ~ t Since P* t is unbiased, the estimate obobtained by replacing DO t by D t is too small on the average except in

Downl



15/26

7 AMERICAN STATISTICAL ASSOCIATION JOURNAL JUNE 1958special cases where D t = DO t with probability one , as has been pointed outmany times e.g., [3] and [23] . Examples such as that of 1.3 show that theresulting bias can be very great, being in fact limited only by the necessitythat the expectation be positive. Since DO t is often unknown while D t isknown, an investigator who is unacquainted with the PL or one of its approxi-mations is all too likely to fall into the trap.Another insidious but instructive characteristic of the illegitimate estimatebased on D t is the fact that it is simply the average of the PL estimates etthose that are defined at t obtained by regarding each item as a sample in itsown right. For a sample of one i tem with observed lifetime h, pet is unity forttl, and for t tl is zero or undefined according as the observation ends withdeath or loss. This is also the RS estimate unless one insists that P* t is unde-fined for t L I the observation limit of the item , even though it may havebeen estimated as zero for smaller values of t.The moral of this discussion is that arithmetic averages of independent PI.estimates from very small samples will tend to be biased. On the other hand,the averaging of independent RS estimates is quite satisfactory, especially ifthe values of N t are used as weights. To avoid bias one must at least assignzero weight when N t =0 thus supplying each estimate with a limit of ob-servation the maximum of those attached to its items beyond which it isregarded as undefined, even though a value of zero may be indicated at someearlier age.

To summarize, the advantages of the PL over the RS are the following:a The observation limits need not be known for items observed to die.b The sampling variance of the PL is usually a lit tle smaller.c The PL has fewer discontinuities, and i ts monotonicity may be comfort-ing.On the other hand, the RS has the following advantages over the PL:a is perfectly unbiased, and hence estimates from manV small samplescan be combined by weighted averaging.b facilitates the estimation of an isolated value of pet .c may be preferred in some cases of dependence between lifetimes andobservation limits see 3.2 below .3.2 Dependence of deaths and losses. The assumption that the full lifetimesI , are independent of the observation limits L. is sometimes violated, as a re-

sult of a change in the population sampled, or the conditions leading to theevent called death or the method of sampling the population. For example, ina study of survival after an operation, a change in surgical technique five yearsbefore the data are analyzed will affect the survival t imes only of those withobservation limit less than five years. When loss is due to unforeseen circum-stances, such as patients moving out of the state, the possibilities for depend-ence are obvious. Merrell [23] and Sartwell [24] have emphasized this point.The estimate of pet from the sample as a whole, whether PL or RS, involves

arbitrary assumptions whose danger is peculiar to the dependent case and henceeasily overlooked. The PI. estimate in effect assumes that for items having an

Downl



16/26

NONPARAMETRIC ESTIMATION 471observation limit L less than t the conditional survival probability P t jP Lis the same as that for items whose observation limits exceed t while the RSestimate makes the same assumption concerning the absolute survival prob-ability Pet itself. In many applications, the PL assumption may seem asplausible as any; on the other hand, the fact that the results are expressedin terms of absolute probabilities may lead one to prefer the RS assumption when there is no feeling to the contrary , if only because its operation is moreeasily visualized. For example, the RS estimate P* t apart from samplingfluctuations) can never fall outside the range of the t rue values for the popula-tions sampled, because it is a weighted average, whereas in unfavorable casesthe PL estimate pet may fall outside this range.

prior observation limits are given for all the items and the sample islarge enough, dependence may be inquired into by grouping the items accordingto these limits, making separate estimates of pet for each group, and comparingthe results. some unanticipated losses occur in advance of the a priori limits,the PL method can be used within each group. The group estimates can thenbe averaged with the initial numbers of items as weights to give somethingsimilar to the RS estimate, that is desired.On the other hand, no dependence can be demonstrated if one s informationis limited to the values of the = min L i , T i and their classification as deathsor losses; in this case the observed rates of death and loss can always be repre-sented by a model in which deaths and losses occur independently, as well asby many models in which this is not so. However, one usually has or can obt ainother information that is more or less useful.

What one would like is to obtain for each item the value of an auxiliaryvariable V which may be either quantitative or qualitative), such that the T iand L; are more nearly independent within subsamples defined in terms of Vthan they are in the sample or population as a whole. This will generally betrue if V is strongly related to the cause of loss, and independent of the cause ofdeath, or vice versa. In this case the subsamples are formed so as to reduce thevariability of the L, or the T i within them. Even if the auxiliary variable doesnot have the desired properties, it may be worthwhile to get an estimate ofpet by a different route for the sake of comparison. Also, the auxiliary variablemay be of interest for its own sake, for example, classification of the items bystarting date would be aimed at detecting a temporal change in the population.As pointed out in 3.1, the procedure in the general case is not satisfactory theestimates depend on very few items.

The innocuous form of dependence that results from testing with replace-ment is considered briefly in Section 7.

4. ACTUARIAL ESTIMATES4.1 Estimates using only n lI t has already been indicated in 2.1) thatactuarial estimates, like the PL, are formed by multiplying together a sequenceof estimates of conditional probabilities of survival through intervals 0, Ul), Ul,U2 Unlike the PL, the actuarial estimates will generally be some-what dependent on the selection of these intervals. t remains to consider how atypical factor p of this sort can be estimated when the number of items n at the

Downl



17/26

472 AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 1958beginning of the interval is known to be depleted by 0 deaths and X losseswithin the interval, but the order in which these occur is not known or used.The PL estimate is denoted by p.As in 2.1, the estimates

n op= nn o n 4a)

4b)

4c)

would be used if all the deaths were known to precede all the losses, and if alllosses preceded all deaths, respectively. Evidently p ~ p p Pis not known,another intermediate value is clearly supplied by-the well-known adjustedobserved estimate cf. [3], [12], [18], [22])

n - A 2 - ap l =n - X/2

This can be recommended for its simplicity even thoughn - X/2 a n - 0/2 - X n - 0 - Xn - A/2 n-o/2 n

Here the second fraction is the corresponding estimate of the probability of notbeing lost if losses are also random events), and the third fraction is the observed probability of escaping both death and loss. The only other estimate inthis section that fails to give equality here is p 4 below.The joint risk estimate cf. [6] and [12]) is

[n - 0 X 5/ HA)p 2) = n 4d)

t is the maximum likelihood estimate when the losses are random, the instantaneous event-rate for losses is a constant times that for deaths, and the onlydata given are the values of n, 0, and The above estimates satisfy the relation r..-::; p 2) -::; p l) ::; p.

One can obtain the PL estimate pfrom any of the foregoing by dividing thegiven interval into smaller intervals no one of which contains both deaths andlosses. Since there are reasons full use of information, absence of arbitraryassumptions, essential uniqueness of the estimate) for preferring small intervalsto large, it seems reasonable to regard p as the standard to which the largeinterval actuarial methods are approximations.

The error incurred in an actuarial estimate of p can be attributed to twosources, namely, the sampling error of p reflecting the fact that 0 is a randomvariable, and the discrepancy between p l and p reflecting the unknown arrangement of deaths and losses within the interval. These sources of error willbe compared by calculating the variance of log pin each case, on the assumptionthat n is large compared with 0 and A and that factors of the form 1 0 n-2can be neglected. On this basis the effect of variation in 0is indicated sufficientlyby the square of the coefficient of variation of p 1 , namely

olen - X/2 n - 0 A/2 4{j 2n - 0 A -2. 4e)

Downl



18/26

NONPARAMETRIC ESTIMATION 473To evaluate the second source of error we note first that the total range ofpossible variation in log p given 0 and A is by 4a

log P - log l = 40A 2n - 0 A -2. 4fHowever, if the arrangement is random, the extreme values may be unlikely.Therefore it will now be assumed that all permutations of the 8 deaths and Alosses are equally likely which happens to be a consequence of the proportionalevent-rates leading to 4d . In this case log p is minus the sum of a randomsample of 0f the 0 1 numbers

log +_1_ log +_1_ ,log + 1 .n n 2 n b A

Sampling theory and the Euler-Maclaurin theorem lead to the approximatevariance40;\ 8 ;\ 1 j3 2n - 0 ;\ 4. 4g

Dividing the sampling variance 4e into the square of half the range 4f ofthe grouping error gives

4hwhich may be either> 1 or < 1. Dividing 4e into the grouping variance 4ggives

;\ 0 + ;\ + 1 j3 2n - 0 ;\ 2, 4iwhich is always O .

The above results suggest that rather large intervals could be used if enoughwas known about the mechanisms of death and loss, and that the principalsource of error is the probable failure of the various permutations of deaths andlosses within the interval to be equally likely, or to fit any other scheme thatmay be assumed. Evidence on this point could be obtained by examining thepermutations that actually occur. For example, this is what is done in using2a . Otherwise one should probably require at the least that 4h be suitablysmall. This still assumes that grouping errors, like the sampling errors, are independent from interval to interval. not, one is thrown back on the ultimate inconservatism, which is to assert only that I I j E j ~ P ~ I I p j

4 2 Estimates using average ages Several estimates will now be consideredthat make use of the individual ages of death and loss, although arbitrary ageintervals continue to be used. The individual ages enter via the average age ofthe 8deaths, and the average age A of the ;\ losses, both measured from the beginning of the age interval under consideration, which is of length h

The first estimate assumes nothing about the losses, but assumes that theinstantaneous death-rate is constant throughout the interval. The total exposure to the risk of death in the interval is then n-o-;\ h+M+;\A, the maximum-likelihood estimate of the death-rate is oj [ n-o-;\ h+M+;\A], and thecorresponding estimate of the survival probability p is

p 3 = exp -o/ [n 0 \ M XA /h] 4j

Downl



19/26

474 AMERICAN STATISTICAL ASSOCIATION JOURNAL JUNE 1958By using p S , Harris, Meier, and Tukey [18] have been able to give an estimation procedure for the more general and more difficult case in which the age ofdeath is never known exactly, but only known to fall in some interval, semiinfinite or finite, bounded by one or two observation points which vary fromitem to item.The estimate p can be derived from p S as well as from p I or p 2 . -though the l imit of the product of the estimates piS for the subintervals is notquite unique, the average value of log p 8 corresponding to 0 deaths occurringat the same age is f l-0 o n - du n - 0= l o g0 1 - u n 4k)in agreement with p= n- 0)In. Here 11 = uh has been regarded as uniformlydistributed between 0 and h. However, in other respects the behavior of p 3does not parallel that of p. the position of one or more of the deaths is shiftedtoward the end of the interval, p S is increased, while pis decreased or else notchanged). Again, p S does not necessarily agree with p in the simple case of nolosses. Finally, the extreme values of p S , obtained by putting I1=A=O or h,fall outside the interval .: p within which p is confined; in fact one can haver


20/26

NONPARAMETRIC ESTIMATION 475primarily on adequately small intervals, and to minimize the computationtime by using simple estimates. This suggests that pO ) be used when the deathsand losses are thought to be arranged at random within the interval, p whenthe interval is chosen so that most of the deaths precede most of the losses, and when losses precede deaths.This completes the material that seems to be most important for applications. The remaining three sections are devoted primarily to mathematicalderivations.

5. MAXIMUM LIKELIHOOD DERIVATION OF THE PLIn accordance with 1.2 the method of maximum likelihood is viewed as a

means for selecting the best fitting distribution from a class of admissible distributions. t will now be shown that if all possible distributions are admitted,the product-limit is the one for whic h the sample likelihood is a maximum.Let T1


21/26

476where

AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 1958

i i l i j l knj = N - I Xi - Oi = Xi Oi or n Tj - 0i=O i= l

in the notation of 1.4. In this form each factor with a fixed index j is maximizedindividually by the binomial estimate pj=pj= nj-oj /nj, in agreement withthe PL in 2a . f Xk>O so that the greatest observed lifetime t* corresponds toa loss, then F t* >0 and the likelihood of the sample is independent of thevalues of pet for t>t*. Thus F t and the maximum likelihood estimate areindeterminate under the same circumstances, and the identity between them iscomplete.

6. MEANS AND VARIANCESIn this section the mean of F t and the variances of Pet and of the meanlife estimates j and p.* are derived. The observation limits are regarded as fixedfrom sample to sample. In principle, unconditional results for random loss

times could be obtained by integration.6.1 The PL estimate F t . Let L1L;: . . . Lk-1be the distinct observation limits that are less than the age t = Lk at which pet is being estimated. Letnj+1=n Lj+0 be the number of survivors observed beyond Lj; Nj+1=Enj+1=N Lj+O P Lj ; Pi=I-Qj=P L j /P Lj_1 ; and O the number of deathsobserved in the interval L j _ 1 L j ] excluding nonzero values of the lef t endpoint, with L= O Then

kF t = Pi with Pi = nj - oj /nj.j= l

6aLet E j denote a conditional expectation for n1 , nj and hence also

1 , OJ_1 fixed. Then EjPj=Ph provided that nj>O. f the condition nj>Ocould be ignored, one could writeEF t = E[h h-1Ekh] = E[h h-1Pk]

= pkE [h Pk-2Ek-1Pk-d 6b= . . . = PkPk-1 P1 = pet ,

so that F t would be an unbiased estimate. The flaw in this demonstration isthe indeterminacy resulting when for some j n j 1 = 0 but F L j >0, an eventwhose probability is bounded by 2d , and which could be obviated in principleby obtaining one or more supplementary observations. In the derivations ofapproximate formulas that follow, any bias that F t may have is neglected.

Both Greenwood [15] and Irwin [19] when giving formulas for the varianceof an actuarial estimate of pet actually treat a special case in which the actuarial estimator and F t coincide. Translated into our notation their common argument is the following.Suppose we consider conditional variances for nj held fixed at the values N j

1 The expression j+0 means losses at jitself have been subtracted off.

Downl



22/26

NONP R METRIC ESTIM TIONThen the P; may be treated as independent quantities and

477

(6c)

I f we ignore terms of order 2 we have as an approximation for the variance(6d)

(6e)

(6f)

This last expression we will refer to as Greenwood s formula. The calculationof the variance with n1, , nk fixed can apparently be justified only on theground that it doesn t matter a great deal; the fixed nj are unrealistic, and inconsistent with the previous assumption of a sample of fixed size and fixedlimits of observation.The authors have therefore rederived (6d) by means of successive conditionalexpectations. The procedure is to verify by induction the approximate relation

q.)Ei+1 [ih t)] h pj2 1 L: Pi+12 Pk2

i ~ i PiEj+1niwhich reduces to (6d) when j = Applying E j to Ge has the effect of replacingj by j provided that one sets

s, p;2 s, [Pi Ejp; ]e.;. E jEi+1ni pj2/E,ni.

wherein pj/ Ei+1ni has been replaced by the ratio of the expectations of numerator and denominator. The fact that pj and Ei+1n, are positively correlated improves the approximation.In the special case in which k=2 (all observation limits less than t are equalto L1) and any indeterminacy is resolved by one supplementary observation,the exact variance of pet can be calculated to be

( [ ( N )2 N [ ]V [P t)] = P2 t) + q2 1 1 _ _ . P - , PINPI N p1P2 N [N, pd (6g)+ N _N ) ( - 2q1N ) ] ) N N [N, pdwhere N is the total sample size; N =N t), the number of items observableto t or beyond; and

[N, pd = 1/E X-1) (6h)with X max Y, 1) and Y distributed binomially , P1 . The methods usedby Stephan [27] show that [N, pd =N P1+0 1) for N large. Thus the term

Downl



23/26

478 AMERICAN STATISTICAL ASSOCIATION JOURNAL, JUNE 1958

(6i)

(6k)

in square brackets is of order 1+O I/N ), and (6g) approaches Greenwood sformula (6d) for large N .6.2 Covariances and mean lifetimes. The method of Irwin [ ] can be usedto derive the variance of j = J; P(t)dt by writing

E jJ.2 = E 1 0 1 0 p(u)P(v)dvdu= 2 i 00 f ooE[P(u)P(V)]dVdU.

u2(U)E h+l Pn+1Pn+ h J== Ph.+lPh+2 PkE [P2 U J (6j)== [P(v)/P(u) ]E[[>2(U)] == P(u)P(v) U(u)],

where by (2h) and (2g)U(u) = f IdP(t) I == [(N - r)(N - r + 1)]-1; N(t)P(t)

the summation is over deaths at ages not exceeding u, and gives the sampleestimate. Thus one has approximatelyV(.u) == 21 0 I, P(u)P(v)U(u)dvdu

= 21 o A(U)P(U)U(U)dU = i A2(U)dU(U),where A(u) =],, P(v)dv. The result is discussed in 2.3.

To obtain analogous but in this case exact) results for the RS estimatesP* (t) and fJ- let u


24/26

NONPARAMETRIC ESTIMATION 4797. CONSISTENCY; TESTING WITH REPLACEMENT

The RS estimate P* t), being binomial, is of course consistent. In view of theapproximate nature of the formula for the variance of Pet), however, a proof ofits consistency is in order. In fact, by giving closer attention to the errors ofthe approximations in the variance derivation, it can be shown that Pet) doeshave the limit pet) in probability provided only that N t)-- > 00 . However, theproof is too lengthy to be given here. Of course, the consistency is obviousenough in the special case in which the number k of conditional probabilities tobe estimated remains bounded as N t)--,> 00.

Thus far the independence of deaths and losses has been assumed. Thequestion of consistency may also be raised in the differing context of testingwith replacement. (Cf. [7], [10], [11], [14].) This is the common life-testingsituation in which a fixed number of items, say v of them are always undertest; when one dies it is replaced by another. The only preassigned constantsare the duration of the test, S, and the number, v, of items on test at all times.The number of losses is then fixed at v, one for each item on test at the conclusion of the experiment, while the number of deaths observed, and hence thetotal number of items, is a random variable. The first v items on test have 8as their limit of observation; if one of these dies at age t, with t00, which implies that either v >OO or 8-- >00 (or both).

The RS estimate will be considered first. Let 1 1, 1 2, , be an infinite sequence of independent lifetimes with survivorship pet) and let Pk t) be theproportion of 1 1, , Tk that exceed t. By the strong law of large numbers,lim Pk t) = pet) with probability one as k- 00 for any specified t. Now the RSestimate P* t) (which exists only for values of t in the interval 0, S) can berepresented as a value of Pk t), with k a random variable. This can be done byassigning the lifetimes 1 1, , 1 , to the v items initially tested. some ofthese items have replacements whose limits of observation 8 - 1 ; (i = 1, ... , v)are not less than t, then succeeding lifetimes 1 ,+1, 1 ,+2, , are assigned tothese replacements. The process is repeated until no more observation intervalsof at least the duration t are left. To prove P* t) consistent, it only remains toprove that for any M, Pr k> M)--,>1 as v8-- > 00 . v > 00 the result is evident. 8-- > 00, then even for v= 1,

Pr k > M = Pr 1 1 + . . . + 1 M 8 - tPr [1 ; (8 - t /M for i = 1, M J (7a)

= {1 - P[ 8 - t)/MllM.As 8-- > the last expression has the limit unity and consistency is proved, provided that P 00 =0 (eventual death is certain).

When 8-- > 00, v is fixed, and P 00 = 0, the consistency of the PL estimatePet) follows from the relation (of, (4a))net i net + v < ret < t) + D - - N t) + D + v net) + vN 7b

Downl



25/26


26/26

NONPARAMETRIC ESTIMATION 481[20] Jablon Seymour, Testing of survival rates as computed from life tables unpublished memorandum June 14, 1951.[ Kahn H. A., and Mantel Nathan Variance of estimated proportions withdrawingin single decrement follow-up studies when cases are lost from observation unpublished manuscript.[ Lit tel l, A. S., Estimation of the T-year survival rate from follow-up studies over a

limited period of time uman Biology 24 1952), 87-116.[23J Merrell, Margaret Time-specific life tables contrasted with observed survivorship,Biometrics 3 1947), 129-36.[24] Sartwell, P. E., and Merrell , M., Influence of the dynamic character of chronic disease on the interpretation of morbidity rates medcan Journal of Public Health 421952), 579-84.[25] Savage, 1. R., Bibliography of nonparametric statistics and related topics, Journalof the American Statistical Association 48 1953), 844-906.[26] Seal, H. L., The estimation of mortality and other deeremental probabilities,

Skandinavisk Aktuarietidskrift 37 1954), 137-62.[27] Stephan F. F., The expected value and variance of the reciprocal and other negativepowers of a positive Bernoullian variate nnals of Mathematical Statistics 61945), 50-61.[28] Wolfowitz, J., Additive partition functions and a class of statistical hypothesesnnals of Mathematical Statistics 13 1942), 247-79.

Downl


Documents

Kaplan, 1958. Nonparametric Estimation From Incomplete Observations