Eight Survey Rules of Thumb - American Statistical · PDF fileAbout Rules of Thumb Why These 8? Each Rule Covered Quickly Some Concluding Remarks About Overstatements ... Eight Survey

Eight Survey Rules of ThumbFritz Scheuren

[email protected]

NORC

Eight Survey Rules of Thumb – p. 1/39

Outline of Presentation



This Brief Background




About Rules of Thumb





Why These 8?





Why These 8?

Each Rule Covered Quickly





Why These 8?


Some Concluding Remarks





Why These 8?



About Overstatements





Why These 8?




Challenge to Practitioners





Why These 8?




Challenge to Practitioners

The Good Stuff


Background Reminders



Original Sampling Paradigm




Taught by Example




Taught by Example

Written Down in 40s thru 60s




Taught by Example


Slow Refinements 70s thru 90s




Taught by Example



Growing Dissatisfaction too




Taught by Example




Consolidate and Move on?




Taught by Example




Consolidate and Move on?

Biemer- Lyberg as Example





Valuable Starting Points




Useful As Historical Record





Form a Checklist





Form a Checklist

Needed as Mnemonics?





Form a Checklist


Reminders of Past Workarounds





Form a Checklist


Reminders of Past Workarounds

A Community Project?


Why These 8?


Why These 8?

Eight is Lucky?


Why These 8?

Eight is Lucky?

Ok, No Good Reason!


Why These 8?

Eight is Lucky?

Ok, No Good Reason!

Sampling v. Nonsampling Tradeoffs


Why These 8?

Eight is Lucky?

Ok, No Good Reason!


Model Based v. Design Based?


Why These 8?

Eight is Lucky?

Ok, No Good Reason!



To Talk about Deming


Why These 8?

Eight is Lucky?

Ok, No Good Reason!



To Talk about Deming


Notes on Nonresponse Presentation

In the slides that follow much has been left unwrittenbecause it was intended to be said. Those spoken wordsare not included here, although not to be too cryptic, Ihave added a few notes to hint at what I was trying to say.



First, the (obviously vast) topic of nonresponse has beenconfined to unit nonresponse. No mention is made ofitem nonresponse, for example. Arguably many pointscarry over nonetheless.



Second, the topic has been treated from an historicalperspective, following the approach I have been taking inmy History Corner series in The American Statistician(TAS). In this connection the November 2004 TAS issuemay be of particular value, as it brings out some of mypoints in more detail.



Third, the formulas provided are more LOGOS thatprescriptions to be used in real life applications. Theywere part of the arguments that the original framers ofthese ideas used but are oversimplified. For example,most of these ideas would be imbedded in pre or poststrata to give them more plausibility and content.



Fourth, in some cases the formula has been associated inthe slides with their author. But this has not always beendone so each of authors of these formulas or rules hasbeen given mention below.



Fifth, the bias/variance tradeoff slide (No.1) was takenfrom the introductory chapters of the HHM (Hansen,Hurwitz and Madow) and Cochran sampling texts. Theirresult has been repeated here to introduce how ourprofessional (over) emphasis on sampling was sold.Those texts carved out the sampling role from the muchlarger data collection issues - all too successfully it mightbe added.



Sixth, the response (and implicitly nonresponse) biasfactors formula (No.2), original with me, was an attemptto succinctly indicate how wrongly our practicecontinues to overemphasize sampling, even 50+ yearslater. We need to add significantly to our records theextensive paradata that is routinely assembled in oursurveys and bring it forward to the client and to futuresurvey developers.



Seventh, the missingness mixture slides (Nos. 3 to 5) aremy interpretation of Rubin’s seminal work onmissingness (Biometrika 1976), where the emphasis ison the ubiquity with which all forms of missing unitnonresponse existing simultaneously in nearly allpractical settings. Our failure to act on this knowledge isa weakness in our practice.



Eighth, the Hansen-Hurwitz subsampling idea fornonresponse is discussed next (No.6). Here the attempt isto illustrate how we could reinterpret their work usingRubin’s idea of the multiple forms of missingness and,thereby, reduce the variance penalty if we have sufficientconfidence in our missingness model. The interviewmode change made as part of the original idea issometimes lost and I regret that I did not say more aboutit last week.



Ninth, the Sekar-Deming capture/recapture applicationto nonresponse (No. 7) concludes the talk, except forsome discussion of implications. Since these ideas arefamiliar in their dual systems incarnation, there is less tosay here that elsewhere.



Tenth, the concluding discussion slides were just my takeon our history Many good suggestions were made forbetter terminology by those present. Several objected toRubin’s use of the phrase "Missing at Random,"abbreviated as MAR. Why not say, instead, they argued,that if we have the right variables present (say in theframe), then we can condition on them and thuseliminate some of the bias.


Eight Rules or Equations



1. Bias versus Sampling Error




2. Response Bias Factors





3. Mixtures of Missingness






4. Data Collection Missingness







5. Data Analysis Missingness








6. Subsamples for Nonresponse









7. “Bias” or Hard-Core Non-Response Rates









7. “Bias” or Hard-Core Non-Response Rates

8. Robust confidence interval estimation





MSE[f(Y )] =

√

σ2

n+ B2

=

√

σ2

n(1 + R2)

(e.g., as shown in the Hansen, Hurwitz and Madow text.)


2. Survey Bias Factors

∫ ∫ ∫

f(Y,M,R,C,D)dM dR dC = g(Y,D)

Implicitly, this formal integration symbolizes that finalsurvey data files usually treat process fixes as complete.Leaving out the paradata concerning these flawscontinues in a way the historical theme in slide No. 1.



In general

m = mMCAR + mMAR + mNMAR

can become specialized at various survey stages






Usually becomesm = mNMAR!

Implicitly, given what is done



Some Alternatives



Some Alternatives

Subsampling Solutions,m = mNMAR



Some Alternatives


Previous Experience, See Example



Some Alternatives



Adaptive Approach, Decide on Fly



Some Alternatives




Enrich the Frame, Make Morem = mMAR



Some Alternatives




Enrich the Frame, Make Morem = mMAR

Use a Composite






Usually becomesm = mMAR!

Again, given what is done



Some Alternatives



Some Alternatives

Noncontact Nonresponsem = mNMAR?



Some Alternatives


Refused Nonresponsem = mMAR!



Some Alternatives



Adaptively Balanced Adjustment



Some Alternatives




Sensitivity Analysis



Some Alternatives




Sensitivity Analysis

Weight Trimming


6. Hansen-Hurwitz Formula

...But Reinterpreted



...But Reinterpreted

y = pRyR + pM yM

where

pR = "responding" fraction of the original sample

pM = remaining "non-responding" fraction of theoriginal sample



...But Reinterpreted (cont’d)



...But Reinterpreted (cont’d)

v(ˆy) ≈ pR

s2R

n+ pM

s2M

vn

+1

n − 1

[

pR(yR − ˆy)2 + pM(yM − ˆy)2]

wherev = sub-sampling rate and other terms, e.g.,s2R

,are just the standard notation



Mixture Example



Mixture Example

Mainly SubsamplemN Noncontacts



Mixture Example


Treat Most (80%?) asmN = mNMAR



Mixture Example



Lightly SubsamplemR Refusals



Mixture Example




If there are good covariates present



Mixture Example





Treat "Bulk" asmR = mMAR + mMCAR



Mixture Example





Treat "Bulk" asmR = mMAR + mMCAR

"Bulk" can sometimes be estimated?


7. Sekar-Deming "Bias" Rates



Employ two samples on similar subjects




Include cases from first survey in second





Unlike Hansen-Hurwitz, obtain both Respondentsand Nonrespondents





Unlike Hansen-Hurwitz, obtain both Respondentsand Nonrespondents

Create a2 × 2 Table where the cells arerr = responded both timesrn = responded first time onlynr = responded second time onlynn = never responded


7. Capture/Recapture Table

[

rr rn

nr nn

]


7. Potential Bias Rate



dd =(rn)(nr)

rr



dd =(rn)(nr)

rr

Then

bb =nn − dd

m

can be considered the potential bias rate


7. Application Experiences



UI Nonprofit Nonresponse Rate = 27 %

UI Nonprofit Potential Bias Rate = 11 %





NSAF Nonresponse Rate = 38 %

NSAF Potential Bias Rate = 15 %





NSAF Nonresponse Rate = 38 %

NSAF Potential Bias Rate = 15 %

NORC Sales Tax Nonresponse Rate = 79 %

NORC Sales Tax Potential Bias Rate = 28 %


Potential Next Steps



These Rules of Thumb obviously will not workeverywhere but have been good starting points inmany practical settings




Some reallocation of resources in survey executionand adjustment seems worthy of consideration insuch settings




Some reallocation of resources in survey executionand adjustment seems worthy of consideration insuch settings

Estimating the missingness mixture fractions is keyhere. One way to do this has been proposed butmore are needed


Provide to End Users and FutureSurvey Designers



Metadata/Paradata should include all relevantcircumstances of the interview




For example number of Calls, who was pesent,Language used, Length of interview, Data on casefrom Frame, etc




For example number of Calls, who was pesent,Language used, Length of interview, Data on casefrom Frame, etc

Nonrespondent cases should also go forward s afinal deliverable, with details like point of Breakoff,Frame characteristics, number and conditionsaround Attempts, etc


Kish-Hess Data Files



Employ Kish-Hess idea of consciously reusingnonrespondents but expand




Keep safe identifiers of both respondents andnonrespondents for possible later use





For bias reduction/analyses, as originally proposed,or for potential bias rate calculations, as describedhere





For bias reduction/analyses, as originally proposed,or for potential bias rate calculations, as describedhere

Could greatly lower long-term costs, whileimproving quality and enhancing credibility


Reanalysis Opportunities



Consider public archiving of the augmented files




Revisit the augmentation regularly, as ourunderstanding of what are the key paradata itemscan grow or change




Revisit the augmentation regularly, as ourunderstanding of what are the key paradata itemscan grow or change

Examine whether partnerships across organizationswould have merit in measuring potential bias rates


Unintended Consequences and Com-plications



Look at statements made to potential respondents,should a sample of them be purposely returned




Examine confidentiality issues that might arise evenif former response status of a case was to bedisclosed




Examine confidentiality issues that might arise evenif former response status of a case was to bedisclosed

Run cognitive tests of these ideas, for potentialrespondents and clients


8. Robust Confidence Interval Esti-mation



If the ratio is less than 1.0, use the average DEFT tomake your adjustment for a given category.




If the ratio is 1.0 to 2.0, use the actual DEFT. Usingthe actual DEFT implies that the calculated estimateof the standard error is the appropriate one to use.





If the ratio is greater than 2.0, subtract the averageDEFT from the actual DEFT and use the remainderto make the adjustment. After you compute the ratioof the actualaDEFT to the average DEFT, takeathefollowing into consideration:















If the ratio is greater than 2.0, subtract the averageDEFT from the actual DEFT and use the remainderto make the adjustment.


THANK YOU...


Documents

Eight Survey Rules of Thumb - American Statistical · PDF fileAbout Rules of Thumb Why These 8? Each Rule Covered Quickly Some Concluding Remarks About Overstatements ... Eight Survey