PHC 6716May 18, 2011

Chris McCarty

CensusCensus - Data collection (or an attempt at data

collection) from every member of a population

Purpose – To know certain characteristics of a population

Example 1 – US census every ten years is a census of households

Example 2 – A survey of all members of the Florida Association of Realtors

Why and when to do a censusThe results of a census are a description of the

population

There are no concerns over inference of the results

It is ideal when the size of the population is relatively small

A census is subject to non-sampling errorSystematically missing the homelessSystematically missing highly mobile people

SurveyData collection (or an attempt at data

collection) from a sample of a population

Surveys are subject to sampling and non-sampling errorSampling error – Failure to capture population

characteristics due to chance

Reasons to do a surveyScenario 1 – Sample designed to estimate the

prevalence of something

Scenario 2 – Sample designed to test the relationship between variables (must represent the range of variables used to test relationships)

Scenario 3 – Both

Example 1 - Florida Health Insurance SurveyClient - Florida Agency for Health Care

Administration (AHCA) and U.S. Health Resources and Services Administration (HRSA)

http://ahca.myflorida.com/Medicaid/quality_management/mrp/Projects/fhis2004/PDF/fhis_comparison_report_aug2005.pdf

Sample designed to be representative of the population (or subgroups) for the purpose of estimating the prevalence of something

FHIS was designed to estimate the rate of the uninsured for Florida, regions of Florida (17), Race and Ethnic subgroups in Florida, Income levels in Florida

DesignRandom Digit Dial Telephone survey

All telephone exchanges in Florida were divided into a set of 85 strata defined by district, income race and ethnicity.

Using census data overlaid to exchanges (GENESYS), initial targets were set

After first wave of completes, targets were readjusted

After second wave, targets were readjusted

Specifics135,976 telephone numbers released

17,435 completed interviews (about 8.4 numbers released per complete)

Approximately 14 minutes per interview

Letter sent to all non-contacts in last months of survey

Result – Percent of Floridians under age 65 who were uninsured in 2004 was 19.2% (up from 16.8% in 1999)

Example 2 – Oral Pain SurveyClient – UF College of Medicine and National

Institutes of Health

Baseline survey with three-month follow-up

Survey designed to capture respondents with particular oral pain symptoms and particular demographic characteristics

Purpose: To understand relationship between demographic characteristics (race and ethnicity) and oral pain symptoms while controlling for intervening variables (income, sex, age)

Specifics2,776 baseline completes out of 59,483 released

RDD sample with disproportionate banks associated with Hispanic and African American households

Quotas for cells combining race, ethnicity and income

The follow-up had 1,006 completes out of 1,726 released.

There was a $15 incentive for the baseline and a $15 incentive for the follow-up

Differences between approachesSurveys estimating the prevalence of

something must either be representative or allow for weighting back to something that is representative

Surveys designed to test relationships must have power (i.e. a full range of values) in variables to be tested

A few definitionsPopulation – The people your research says you are

interested in studying

Survey Mode – The process used to collect data from the population

Sample Frame – A list that represents the population and allows you to draw a sample to use with your selected mode

Non-sampling error – Error associated with collecting the data

Sampling error – Error associated with pulling the sample

Defining the populationResearch question suggests population

GeographyDemographic characteristicsTime frame

Examples: 1.Are Florida HMO members satisfied with

their service?2.Do Hispanic migrants get breast cancer

screenings?3.Does obesity in children lead to diabetes?

Survey ModesFace-to-face

Mail

Telephone

Web

Face-to-face – How to do itTypically cluster sampling (unless geography

is small)Use Census tracks and blocks as sample frame to

select an area, then pick every nth householdMake a map of an area as sample frame then pick

every nth householdDepending on population can also use lists as

sample frameTypically make at least three return visits at

different times of the day and weekCan be done with paper and pencil or

computer

Face-to-faceAdvantages and DisadvantagesAdvantages

High response ratesLower levels of satisficing (offering responses that satisfy

interviewer but are not a true representation of fact or opinion)

Higher confidence in respondent selectionUse of show cards and other visual aidsCan usually do longer interviews

DisadvantagesMost expensiveMay be less representative due to compromises in sampling

strategyDepending on population, may be dangerous to interviewersDifficult to maintain interviewing staff

Face-to-face examples Post election survey in Ghana (1997)

Survey of UF students regarding hookah use (2010)

Post election survey in GhanaQuestion: Did Ghanaians think the 1996 elections

were honest?

Sample frame – Polling stations using voter registration rather than Census

Ghana has 10 regions, and each received at least 220 of a total of 2300 interviews

Within each region we distributed a clusters of 10 interviews

Distribution of completed interviews

Region Frequency Percent of Sample

Percent of registered voters

Weight

Ashanti 270 11.7 17.2 1.47

Brong Ahafo 220 9.6 9.8 1.02

Central 220 9.6 8.3 0.86

Eastern 220 9.6 11.4 1.19

Greater Accra 270 11.7 16.9 1.44

Northern 220 9.6 8.7 0.91

Upper East 220 9.6 4.7 0.49

Upper West 220 9.6 2.9 0.30

Western 220 9.6 10.4 1.08

Volta 220 9.6 9.7 1.01

Ghana 2300 100 100 NA

ResultFigure 1. Percent who felt elections were "somewhat dishonest" or

"very dishonest" by region.

0

5

10

15

20

25

Ash

anti

Bro

ng A

hafo

Cen

tral

Eas

tern

Gre

ater

Acc

ra

Nor

ther

n

Upp

er E

ast

Upp

er W

est

Wes

tern

Vol

ta

Gha

na

Region

Per

cent

Hookah Survey BackgroundHookah use among college students is estimated to be

between 10-11%

Universities often rely on Web surveys of students, often e-mailing all students listed by the Registrar

Students are therefore increasingly inundated with e-mails

In the past UF has warned students not to respond to unsolicited e-mail

We proposed a face-to-face survey of 1,000 students

Hookah survey methodFive locations on campus

1. Plaza of the Americas2. Turlington Hall3. West Campus Recreation Center4. Communicore Building5. Reitz Union

Rotated times of days and days of week at each siteTables set up with laptops and a 10 minute CATI

surveyInterviewer offered every 10th person to walk by a

$5 gift card to complete

Hookah Survey ResultsA total of 1,203 completed interviews

Race and sex were weighted, but were not far off Registrar characteristics

10.9% (131) reported current hookah use, approaching the current cigarette use rate of 11.7%

More students have tried hookah (45.4%) than cigarettes (40.3%) or any other form of tobacco

Mail – How to do itAlways use lists as sample frames Usually have return envelope with stamp or meter

where you pay if sentCan do drop-off (has face-to-face limitations)Typically do multiple mailings or post card remindersOften include token incentives in envelopeReturned surveys are sometimes double enteredCan save on costs by outsourcing printing and mailing Can be personalized with signaturesFedX, Priority Mail and First Class more noticeableMay want to hide respondent identifier inside envelopeCan do scannable forms

Mail: Advantages and DisadvantagesAdvantages

Can be less expensive May be better for certain sensitive questions Can include show cards or other visual aids Sometimes is the only choice given available sample frames

Disadvantages Often lower response rates than face-to-face and phone Takes longer to finish survey process Little control over respondent selection Respondents often leave information missing or write in their

own response categories (effectively missing) Limitations with skip logic and use of previous answers in

latter part of questionnaire P.O boxes often not included in sample when overlaying

geography

Mail - ExamplesWater Management District

Water Management SurveyPurpose: Measure household characteristics

and perceptions of water use

Mail out of 7,200 surveys based on utility bill data (address only accurate contact)450 for each of 16 participating water utilities

Double data entry

Water Management SurveyThree stages:

1. Advance Letter one week before survey. Included 1-800 number for questions

2. Survey package $1 incentive for about 1,500 lower income

respondents Self addressed envelope metered to charge upon

receipt Packets in Miami-Dade received packet in English

and Spanish

3. Thank you/Reminder postcard

Telephone – How to do itListed Sample

Listed sample often comes from phone directory Advantage is less dialing Disadvantage is biased phone coverage (nationally unlisted numbers

may be as much as 30%, and 50% in some urban areas)Lists from member files or other databases (This is most of

what we do)Random Digit Dial (RDD)

Telephone numbers made up using information on released banks (a bank is defined by Area code + Prefix + first two digits of suffix)

Not all banks are releasedThey tend to cluster (Waxberg sample)Can have phone numbers purged of businesses and charitiesZero, 1-plus, 2-plus banks

Telephone – How to do it (continued)Predictive dialer – A file server that dials calls

and diverts interview to person when answer detected (responsible for pause)

Sample management software WincatiBlaisemrInterview CATI (SPSS)

Survey analysis software (SUDAAN from RTI)

Telephone: Advantages and DisadvantagesAdvantages

High response rateFastAllows for complex skip logic and use of previous

answers in latter part of surveyRelatively high coverage (about 95% nationally have

phones)More control over respondent selectionComplex sample managementImmediate data entry

DisadvantagesFalling response rates (telemarketing, caller ID, cell

phones)No show cards or visual aidsWith some populations there is no viable frame

Do Not Call ListsNational Do Not Call List (www.donotcall.gov)

Some states (http://www.the-dma.org/government/donotcalllists.shtml)

Surveys and charities are exempt

Respondent usually does not know that

Telephone - ExamplesMonthly consumer confidence survey

HMO Report Card

Monthly CCI SurveyPurpose: Predict Florida consumer spending using index

Field time constrained to one month

Used to be one sample of 5,000 RDD numbers in a month and 500 completed interviews

Changed to two, two-week surveys with 2,600 RDD numbers released and 250 completes

Numbers are released proportionate to households by county with post-weighting for disproportionate coverage

Comparison of Florida and U.S. Consumer Sentiment

April release weighting by age

HMO Report CardPurpose: Measure and publish customer satisfaction

using CAHPS for each Medicaid HMO in Florida

Listed sample pulled from AHCA database for customers who have been in plan for at least 6 months

Attempt 300 completed interviews from each plan for Adults and for Children

Set of indicators published on AHCA web site: http://www.floridahealthfinder.gov/HealthPlans/Compare.aspx

Web – How To Do ItMany online vendors, but they often only

provide questionnaire authoring and storage, little sample management (e.g. Survey Monkey)

Costs are (in my opinion) inflated

Ideal for certain populations

Web: Advantages and DisadvantagesAdvantages

Typically inexpensive (at least it should be)Data are automatically entered and edited upon entryMaximum versatility in the use of visual aids and audioLess satisficing for some sensitive questions

DisadvantagesVery low response ratesIncomplete and biased coverage for household surveys

(only about 75% of households versus 95% for phones)No RDD version for e-mails, lack of comprehensive listsMay be combined with phone or mail to be effective

Web Example – Web of Science

Objective – Determine if co-authorship on the Web of Science is a method for the transmission of scientific innovation

Method – Conduct survey with representative sample of authors on the Web of Science

ProcedureWe began by downloading all unique author/affiliation

combinations from the Web of Science for 2006 – a total of 3,004,946 unique records (one scientist for every 2,181 people in the world)

We removed all records where the affiliation contained the strings univ, sch, or coll. This left 1,084,833 records

These records were randomized and the first 20,000 were exported and an attempt was made to find an e-mail for each record

We found 7,962 which were loaded into a web survey

Estimate of proportion working in non-academic settingOf the 7,962 e-mail addresses sent out we

received 747 (9.4%) completed surveys and indicated they had published an article

We estimate that 683,444 authors, or 23%, do not work at a college or university

Of those respondents working in a non-academic setting, nearly 72% consider themselves an academic

Common sources of listsTelephone numbers and households listed in

telephone directoryCan pull national sampleUnlisted numbers vary a lot by geographic area and

respondent characteristicsDrivers licenses from state Department of Motor

VehiclesMust select samples by state, and states vary in laws

regarding drivers licensesData may be old as people move without informing

Department of Motor VehiclesNor every one drives and there are biases (old and

young, people in urban settings with public transportation and high insurance costs)

Common sources of lists (continued)Voter Registration

Potentially more updated than driver’s license databaseNot everyone votes – potentially very biased unless survey

concerns potential votersLists from behavioral surveys and credit card

evaluationUsually expensiveCan often select people with particular characteristics (e.g.

smokers)Potentially biased based on source

Member and User Lists such as patient records, HMO membership, recipients of Temporary Assistance for Needy Families (TANF)Source is often variable in maintaining records (e.g. HMOs

do not have common database practices for recording membership data)

Companies that supply sampleMarketing Systems Group – GENESYS

Survey Sampling

Affordable Sampling

Telephone survey sample options and costsRDD with no filtering – $.04/record ($300

minimum)

RDD with business purging from yellow pages – $.05/record

RDD with business purge and attended dialing using automated detection – $.09/record

Experian Behavior Bank – $.35/record

Reading an RDD coverage report2000 Census Tract/BG Coverage Report GENESYS Sampling Systems

Market: FL-AA Database Version: V2004-2Date/Time: 9-JUL-2004 10:51:45.14 OSLO Households Excluded================================================================================

IN AREA NON-COVERAGE ========================== ========================== TOTAL CUMULATIVE CUMULATIVEEXCHANGE LISTED HH LHH INC INC COV LHH INC INC COV======== ========= ======== === === === ======== === === ===

3594713 386523 11 11 100 3208190 89 89 100

305503 2 2 100 100 0 0 0 0 0 904244 2 2 100 100 0 0 0 0 0 904457 2 2 100 100 0 0 0 0 0 561880 2 2 100 100 0 0 0 0 0 850718 1 1 100 100 0 0 0 0 0 813383 1 1 100 100 0 0 0 0 0 305328 1 1 100 100 0 0 0 0 0 850310 1 1 100 100 0 0 0 0 0 786328 1 1 100 100 0 0 0 0 0 954301 1 1 100 100 0 0 0 0 0 850220 1 1 100 100 0 0 0 0 0 954241 1 1 100 100 0 0 0 0 0 954809 1 1 100 100 0 0 0 0 0 321319 1 1 100 100 0 0 0 0 0 850353 1 1 100 100 0 0 0 0 0 863260 1 1 100 100 0 0 0 0 0 954550 1 1 100 100 0 0 0 0 0 850260 1 1 100 100 0 0 0 0 0 850856 889 853 96 96 0 36 4 4 0 904354 1470 1349 92 93 1 121 8 7 0 904356 1210 1112 92 93 1 98 8 7 0 305749 12 11 92 93 1 1 8 7 0 904355 1584 1439 91 92 1 145 9 8 0 904350 128 116 91 92 1 12 9 8 0 904353 1369 1232 90 92 2 137 10 8 0 904359 362 326 90 92 2 36 10 8 0 904301 69 62 90 92 2 7 10 8 0 904598 422 375 89 92 2 47 11 8 0 904475 346 309 89 91 2 37 11 9 0 904357 62 55 89 91 2 7 11 9 0 904665 44 39 89 91 2 5 11 9 0

Companies that do most large federally funded surveysWestatAbt AssociatesMathematicaResearch Triangle Institute (RTI)ORC MacroNational Opinion Research Center

(NORC)Institute for Social Research (ISR) –

University of Michigan