Canadian Community Health Survey Cycle 1.1 Overview of methodological issues and more

Canadian Community Health SurveyCanadian Community Health Survey

Cycle 1.1Cycle 1.1

Overview of methodological issues and more...Overview of methodological issues and more...

Presentation OutlinePresentation Outline

Sample DesignSample Design–Target population, sample allocation and framesTarget population, sample allocation and frames

–Sampling strategies, oversampling of sub-populationsSampling strategies, oversampling of sub-populations

–Data collection, response ratesData collection, response rates

–ImputationImputation

–WeightingWeighting

Sampling errorSampling error–Sampling variability guidelinesSampling variability guidelines

–Variance estimation: Variance estimation: BootstrapBootstrap re-sampling technique re-sampling technique

–CV look-up tablesCV look-up tables

AnalysisAnalysis–ExamplesExamples

–How to use the How to use the BootvarBootvar programs programs

CCHS - Cycle 1.1 CCHS - Cycle 1.1 Health Region-level survey Health Region-level survey

Main objective Main objective –Produce timely cross-sectional estimates for Produce timely cross-sectional estimates for 136 health 136 health regionsregions

Target populationTarget population–individuals living in private occupied dwellings aged 12 years old or overindividuals living in private occupied dwellings aged 12 years old or over

–Exclusions:Exclusions: those living on Indian Reserves and Crown Lands, residents those living on Indian Reserves and Crown Lands, residents of institutions, full-time members of the Canadian Armed Forces and of institutions, full-time members of the Canadian Armed Forces and residents of some remote areasresidents of some remote areas

CCHS 1.1 covers ~98% of the Canadian population CCHS 1.1 covers ~98% of the Canadian population

CCHS - Sample Allocation to Provinces CCHS - Sample Allocation to Provinces

ProvProv PopPop # of # of 1st Step1st Step 2nd Step2nd Step TotalTotal

SizeSize HRs HRs 500/HR500/HR X-propX-prop SampleSample

NFLD 551K 6 *2,780 1,230 4,010

PEI 135K 2 1,000 1,000 2,000

NS 909K 6 3,000 2,040 5,040

NB 738K 7 3,500 1,650 5,150

QUE 7,139K 16 8,000 16,280 24,280

ONT 10,714K 37 18,500 23,760 42,260

MAN 1,114K 11 5,500 2,500 8,000

SASK 990K 11 *5,400 2,320 7,720

ALB 2,697K 17 *8,150 6,050 14,200

BC 3,725K 20 10,000 8,090 18,090

CAN 29,000K 133 65,830 64,920 130,750

* The sampling fraction in some small HRs was capped at 1 in 20 households

CCHS - Sample Allocation to Health Regions CCHS - Sample Allocation to Health Regions

Pop. Size # of Mean

Range HRs Sample Size

Small less than 75,000 41 525

Medium 75,000 - 240,000 60 900

Large 240,000 - 640,000 25 1,500

X-Large 640,000 and more 7 2,500

CCHS - Sample Allocation to Territories CCHS - Sample Allocation to Territories

Population Sample

Yukon 25,000 850

NWT 36,000 900

Nunavut 22,000 800

CCHS - Sample Frame CCHS - Sample Frame

CCHS sample selected from three frames: CCHS sample selected from three frames:

•Area frame (Labour Force Survey structure)Area frame (Labour Force Survey structure)

•RDD frame of telephone numbers (Random Digit Dialling)RDD frame of telephone numbers (Random Digit Dialling)

•List frame of telephone numbersList frame of telephone numbers

Three frames are needed for CCHS for the following reasons:Three frames are needed for CCHS for the following reasons:

1. To yield the desired sample sizes in all health regions1. To yield the desired sample sizes in all health regions

2. Have a telephone data collection structure in place to quickly address 2. Have a telephone data collection structure in place to quickly address provincial/regional requests for buy-in sample and/or content at any point in provincial/regional requests for buy-in sample and/or content at any point in timetime

3. Optimize collection costs3. Optimize collection costs

Area frame - Sampling of householdsArea frame - Sampling of households

83% of CCHS sampled households83% of CCHS sampled households Multistage stratified cluster sample designMultistage stratified cluster sample design

Stratum #1

Stratum #2

#1: Each health region is divided into strata

#2: Clusters selected within strata (PPS sampling) (1st stage)

#3: Dwellings selected within clusters (2nd stage)

RDD frame of telephone numbers RDD frame of telephone numbers Sampling of households Sampling of households

Elimination of non-working banks methodElimination of non-working banks method– 7% of CCHS sampled households7% of CCHS sampled households

– Telephone bank: area code + first 5 digits of a 7-digit phone #Telephone bank: area code + first 5 digits of a 7-digit phone #

1- Keep the banks with at least one valid phone #1- Keep the banks with at least one valid phone #

2- Group the banks to encompass as closely as possible the health region areas - RDD strata2- Group the banks to encompass as closely as possible the health region areas - RDD strata

3- Within each RDD stratum, first select one bank at random and then generate at random one number between 00 and 993- Within each RDD stratum, first select one bank at random and then generate at random one number between 00 and 99

4- Repeat the process until the required number of telephone numbers within the RDD stratum is reached4- Repeat the process until the required number of telephone numbers within the RDD stratum is reached

List frame of telephone numbers List frame of telephone numbers Sampling of households Sampling of households

Simple random sample of telephone numbersSimple random sample of telephone numbers– 10% of CCHS sampled households10% of CCHS sampled households

– Telephone companies’ billing address files and Telephone Infobase (repository of phone directories)Telephone companies’ billing address files and Telephone Infobase (repository of phone directories)

1- Create a list of phone numbers1- Create a list of phone numbers

2- Stratify the phone numbers by health region using the residential postal codes 2- Stratify the phone numbers by health region using the residential postal codes

3- Select phone numbers at random within a health region3- Select phone numbers at random within a health region

4- Repeat the process until the required number of telephone numbers is reached4- Repeat the process until the required number of telephone numbers is reached

CCHS - Sampling of persons CCHS - Sampling of persons

Area frameArea frame Simple random sample (SRS) of one person aged 12 years of age or Simple random sample (SRS) of one person aged 12 years of age or older (82% of households)older (82% of households) SRS sample of two persons aged 12 years of age or older (18%) SRS sample of two persons aged 12 years of age or older (18%)

RDD / List framesRDD / List frames SRS sample of one person aged 12 years of age or older SRS sample of one person aged 12 years of age or older

CCHS - Sampling of persons CCHS - Sampling of persons

Age 1996 LFS * CCHSgroup Census sample simulated

(all persons) sample ( only 1 person)

12-19 13.2 13.7 8.5

20-29 16.4 14.4 14.3

30-44 30.8 28.7 29.1

45-64 25.8 28.0 27.9

65 + 13.8 15.2 20.2

* averaged distribution over 100 repetitions using the May 99 LFS sample* averaged distribution over 100 repetitions using the May 99 LFS sample

CCHS - Representativity of sub-populations CCHS - Representativity of sub-populations

To address users’ needs, two sub-population groups needed larger effective sample sizes:

Youths (12-19 years old) –Decision > Oversample youths by selecting a second person (12-19) in some households based on their composition

Elderlies (65 years old and +)–Decision > Do not oversample - let the general sample selection process address the issue by itself

Sampling strategy based on household compositionSampling strategy based on household composition

Number of persons aged 20 or overNumber of persons aged 20 or over

NumberNumber 00 1 1 2 2 3 3 4 4 5+ 5+of 12-19of 12-19

00 - A A A A B

11 A A C C C B

22 A C C C C C

3+3+ A C C C C C

A:A: Simple random sample (SRS) of one person aged 12+ Simple random sample (SRS) of one person aged 12+

B: B: SRS of two persons aged 12+SRS of two persons aged 12+

C:C: SRS of one person in the age group 12-19 SRS of one person in the age group 12-19 andand SRS of one person 20+ SRS of one person 20+

CCHS - Sample Distribution after OversamplingCCHS - Sample Distribution after Oversampling

Age 1996 * CCHS * CCHSgroup Census simulated simulated

sample sample( only 1 person) ( some 2 persons)

12-19 13.2 8.5 14.920-29 16.4 14.3 13.130-44 30.8 29.1 28.145-64 25.8 27.9 26.365 + 13.8 20.2 17.6

* averaged distribution over 100 repetitions using the May 99 LFS sample* averaged distribution over 100 repetitions using the May 99 LFS sample

CCHS - Initial data collection planCCHS - Initial data collection plan

12 monthly samples12 monthly samples 12 collection months + 112 collection months + 1

Area frameArea frame

CAPI CAPI STC field interviewersSTC field interviewers targeted response rate: 90%targeted response rate: 90% anticipated vacancy rate: 13%anticipated vacancy rate: 13%

(09 / 2000 - 08 / 2001) + 09 / 2001(09 / 2000 - 08 / 2001) + 09 / 2001

RDD / List framesRDD / List frames

CATI CATI STC call centresSTC call centres targeted response rate: 85%targeted response rate: 85% telephone hit rate: 15-60%telephone hit rate: 15-60%

CCHS data collection - Observed situationCCHS data collection - Observed situation

Field interviewersField interviewers– workload exceeded field staff capacityworkload exceeded field staff capacity

Call centresCall centres– new collection infrastructurenew collection infrastructure

– unequal allocation of work among call centresunequal allocation of work among call centres

Descriptive paper:Descriptive paper:– « Preventing nonresponse in the Canadian Community Health « Preventing nonresponse in the Canadian Community Health

Survey », Survey », Y. Béland, J. Dufour, and M. Hamel. 2001, Hull, Statistics Y. Béland, J. Dufour, and M. Hamel. 2001, Hull, Statistics Canada XVIIICanada XVIIIthth International Symposium. International Symposium.

CCHS - Final response ratesCCHS - Final response rates

FieldField Call centresCall centres TotalTotalNFLDNFLD 86.686.6 89.389.3 86.886.8

PEIPEI 87.787.7 82.682.6 84.784.7

NSNS 88.888.8 89.389.3 88.888.8

NBNB 88.488.4 92.492.4 88.588.5

QUEQUE 85.785.7 84.884.8 85.685.6

ONTONT 82.882.8 79.579.5 82.082.0

MANMAN 90.090.0 85.085.0 89.589.5

SASKSASK 87.087.0 85.485.4 86.886.8

ALBALB 85.285.2 84.984.9 85.185.1

BCBC 83.983.9 86.786.7 84.784.7

YUKYUK 79.379.3 95.695.6 82.782.7

NWTNWT 89.689.6 85.485.4 89.289.2

NUNNUN 66.366.3 34.634.6 62.562.5

CANCAN 85.185.1 83.183.1 84.784.7

CCHS - Proxy interviewsCCHS - Proxy interviews

DefinitionDefinition: When another person in the household responds: When another person in the household responds

to the survey on behalf of the selected person in the sampleto the survey on behalf of the selected person in the sample Acceptable in the following cases:Acceptable in the following cases:

– Out of the country for a long period of timeOut of the country for a long period of time

– Mental or physical state of healthMental or physical state of health

– Language barrierLanguage barrier

Usually, 2 to 3 % of proxy respondentsUsually, 2 to 3 % of proxy respondents Because of the field problems: Because of the field problems: 6,3%6,3%

– Higher rate in some health regions, for men and younger respondentsHigher rate in some health regions, for men and younger respondents

Major consequence : one third of the questionnaire is missingMajor consequence : one third of the questionnaire is missing

– Personal or sensitive type of questions not askedPersonal or sensitive type of questions not asked

Solution : imputationSolution : imputation

Modules for proxy and non-proxyModules for proxy and non-proxy

AlcoholAlcohol Chronic conditionChronic condition Exposure to second hand smokeExposure to second hand smoke Food insecurityFood insecurity General health (Q1, Q2 and Q7)General health (Q1, Q2 and Q7) Health care utilizationHealth care utilization Health Utility Index (HUI)Health Utility Index (HUI) Height / Weight (Q2 and Q3)Height / Weight (Q2 and Q3) InjuriesInjuries Restriction of activitiesRestriction of activities Smoking Smoking Tobacco alternativesTobacco alternatives

Two-week disabilityTwo-week disability Household composition & housingHousehold composition & housing IncomeIncome Labour forceLabour force Socio-demographic characteristics Socio-demographic characteristics AdministrationAdministration

Drug use (optional)Drug use (optional) Home care (optional)Home care (optional)

Modules for non-proxy onlyModules for non-proxy only Alcohol dependence / abuseAlcohol dependence / abuse Blood pressure checkBlood pressure check BreastfeedingBreastfeeding Contacts with mental health Contacts with mental health

professionalsprofessionals MammographyMammography Fruit & vegetable consumptionFruit & vegetable consumption General health (Q3-Q6, Q8-Q10)General health (Q3-Q6, Q8-Q10) Height / Weight (Q4 only)Height / Weight (Q4 only) PAP smear testPAP smear test PSA testPSA test Physical activitiesPhysical activities Patient Satisfaction**Patient Satisfaction**

Breast examinationsBreast examinations Breast self examinationsBreast self examinations Changes made to improve healthChanges made to improve health DepressionDepression

Dental visitsDental visits DistressDistress Driving under influenceDriving under influence Eye examinationsEye examinations Flu shots*Flu shots* MasteryMastery MoodMood Physical check-up Physical check-up Sedentary activitiesSedentary activities Self-esteemSelf-esteem Sexual behavioursSexual behaviours Smoking cessation aidsSmoking cessation aids Social supportSocial support SpiritualitySpirituality Suicidal thoughts and attemptsSuicidal thoughts and attempts Use of protective equipmentUse of protective equipment Work stressWork stress

Imputation Strategy - 5 passesImputation Strategy - 5 passes

1st pass: Health prevention modules1st pass: Health prevention modules

– 3 modules imputed completely3 modules imputed completely

– 6 modules imputed partially (some questions only)6 modules imputed partially (some questions only)

– 2 modules not imputed2 modules not imputed 2nd pass: Mental health modules2nd pass: Mental health modules

– 6 modules imputed completely6 modules imputed completely

– 7 modules not imputed7 modules not imputed 3rd pass: Sexual behaviours3rd pass: Sexual behaviours 4th pass: Fruit and vegetable consumption4th pass: Fruit and vegetable consumption 5th pass: one question in the module height and weight5th pass: one question in the module height and weight

Note that some modules andNote that some modules and/or questions /or questions are not imputed (such as are not imputed (such as physical activity, distress, work stress, time since last flu shot, etc.)physical activity, distress, work stress, time since last flu shot, etc.)

Imputation StrategyImputation Strategy

Strategy applied at each imputation passStrategy applied at each imputation pass

1. Create imputation classes1. Create imputation classes– Usually: Province X Sex X Age groups X FiltersUsually: Province X Sex X Age groups X Filters

– The donor has to be in the same imputation class as the recipientThe donor has to be in the same imputation class as the recipient

– Minimum donor rule : donors / (donors + recipients) >= 60%Minimum donor rule : donors / (donors + recipients) >= 60%

2. Identify a list of matching variables2. Identify a list of matching variables

3. Assign a weight to each matching variable3. Assign a weight to each matching variable– Default weight = 1, sometimes weight = 2, 3 or more.Default weight = 1, sometimes weight = 2, 3 or more.

4. Find the nearest donor4. Find the nearest donor- Highest Total Weighted Match- Highest Total Weighted Match

- If more than one possible donor, select one randomly from them- If more than one possible donor, select one randomly from them

- No imputation if no donor over a minimum number of matches- No imputation if no donor over a minimum number of matches

CCHS - Weighting and EstimationCCHS - Weighting and Estimation

Estimation relates sample back to population

MUST use weights in calculation of estimates to correctly draw conclusions about population of interest

Sampling weight is related to the probability of selecting a person in the sample

Persons are selected with unequal probabilities therefore have varying weights

CCHS - Weighting and EstimationCCHS - Weighting and EstimationThree separate weighting systems:

–Area frame design

–RDD frame design

–List frame design

Several adjustments– non-response (household and person)

– seasonal factor

– etc...

Integration of the two weighting systems based on design effects and sample sizes ( n / deff )Calibration using a one-dimensional poststratification adjustment of ten age/sex poststrata within each health regionVariance estimation : bootstrap re-sampling approach

–set of 500 bootstrap weights for each individual

Weighting & EstimationWeighting & Estimation

Integration|

Seasonal effect|

Post Stratification|

Final CCHS weight

Area frame Telephone frame

Initial weight (dwelling level) Initial weight (dwelling level)| |

Sampling adjustments Uncoverage of List frame|

Remove out-of-scope units Remove out-of-scope units|

Household nonresponse Household nonresponse|

No phone lines|

# of people in hhld (person wgt) # of people in hhld (person wgt)| |

Person level nonresponse Person level nonresponse|

Multiple phone lines|

Final Area weight Final Telephone weight







No phone lines|





Initial weight: Inverse of the probability of being selectedInitial weight: Inverse of the probability of being selected

Weighting & EstimationWeighting & Estimation Household nonresponse: Distribute weight of nonresponding Household nonresponse: Distribute weight of nonresponding

households to responding oneshouseholds to responding ones– Using “nonresponse classes such as HR, collection period and Using “nonresponse classes such as HR, collection period and

urban, rural/urban)urban, rural/urban)






No phone lines|





Weighting & EstimationWeighting & Estimation No phone lines: No coverage of hhlds without a phone line. Weights No phone lines: No coverage of hhlds without a phone line. Weights

are “boosted” by a certain rate (specific to each HR)are “boosted” by a certain rate (specific to each HR)

– Rates of “no phone lines” calculated using area frame dataRates of “no phone lines” calculated using area frame data






No phone lines|





Weighting & EstimationWeighting & Estimation # of people in hhld: Convert the hhld-level weight into a person-level # of people in hhld: Convert the hhld-level weight into a person-level

weight (multiply by the number of people)weight (multiply by the number of people)

– Depends on the # of people selected (1 or 2), and their ageDepends on the # of people selected (1 or 2), and their ageArea frame Telephone frame





No phone lines|





Weighting & EstimationWeighting & Estimation Person level nonresponse: Redistribute the weight of selected person Person level nonresponse: Redistribute the weight of selected person

who did not respond to the ones who respondedwho did not respond to the ones who responded

– Using classes (age, sex, # person selected, collection period, etc)Using classes (age, sex, # person selected, collection period, etc)






No phone lines|





Weighting & EstimationWeighting & Estimation Multiple phone lines: More phone lines = higher probability of being Multiple phone lines: More phone lines = higher probability of being

selectedselected

– weight divided by the number of residential phone linesweight divided by the number of residential phone linesArea frame Telephone frame





No phone lines|





Weighting & EstimationWeighting & Estimation Final weight: Each frame’s final weight is each representative of the Final weight: Each frame’s final weight is each representative of the

total population. To create a single set of weights, they are combined total population. To create a single set of weights, they are combined through “Integration”through “Integration”






No phone lines|






Integration: Combine the 2 sets of weights into one single set of Integration: Combine the 2 sets of weights into one single set of weightsweights

– Based on sample size and design effect of each frameBased on sample size and design effect of each frame



Integration|

Seasonal effect|

Post-stratification|

Final CCHS weight


Seasonal effect: Adjust weights so that each season contains 25% of the total Seasonal effect: Adjust weights so that each season contains 25% of the total populationpopulation– Based on the collection periodBased on the collection period

(sept-nov / dec-feb / mar-may / june - aug)(sept-nov / dec-feb / mar-may / june - aug)



Integration|

Seasonal effect|


Final CCHS weight




Integration|

Seasonal effect|


Final CCHS weight

Post-stratification: Ensure the sum of weights matches the estimated Post-stratification: Ensure the sum of weights matches the estimated population projections in each HR, for 10 age-sex groupspopulation projections in each HR, for 10 age-sex groups– 12-19, 20-29, 30-44, 45-64 and 65+ crossed with two sexes12-19, 20-29, 30-44, 45-64 and 65+ crossed with two sexes




Integration|

Seasonal effect|


Final CCHS weight

Final CCHS weight: Final weight present on the CCHS Final CCHS weight: Final weight present on the CCHS master file master file

CCHS - Special WeightsCCHS - Special Weights

For various reasons, many other weights are produced

– Quarter 4 special weight

– PEI special weight

– Share weights (master, Q4 and PEI special)

– Link weights (master, Q4 and PEI special)

Sampling ErrorSampling Error

Difference in estimates obtained from a sample as Difference in estimates obtained from a sample as compared to a censuscompared to a census

The extent of this error depends on four factors:The extent of this error depends on four factors:– sample sizesample size

– variability of the characteristic of interestvariability of the characteristic of interest

– sample designsample design

– estimation method estimation method

Generally, the sampling error decreases as the size of the Generally, the sampling error decreases as the size of the sample increasessample increases

Sampling ErrorSampling Error Measures of precision associated to an estimateMeasures of precision associated to an estimate

– Variance Variance

– Standard deviation (square root of the variance)Standard deviation (square root of the variance)

– 95% confidence interval (estimate ± 1.96 x standard deviation)95% confidence interval (estimate ± 1.96 x standard deviation)

– Coefficient of variationCoefficient of variation

• Standard deviation of estimate x 100% / estimate itself Standard deviation of estimate x 100% / estimate itself

• CV allows comparison of precision of estimates with different scalesCV allows comparison of precision of estimates with different scales

– Examples:Examples:

• 24% of population are daily smokers, std dev. = 0.00324% of population are daily smokers, std dev. = 0.003

> CV=0.003/0.24 x 100%=> CV=0.003/0.24 x 100%=1.25%1.25%

> 95% CI: 0.240 ± 1.96 x 0.003 : > 95% CI: 0.240 ± 1.96 x 0.003 : {0.234 ; 0.246 }{0.234 ; 0.246 }

Sampling Variability GuidelinesSampling Variability Guidelines

Type of estimateType of estimate CVCV GuidelinesGuidelines

AcceptableAcceptable 0.0-16.5 0.0-16.5 General unrestricted releaseGeneral unrestricted release

MarginalMarginal 16.6-33.3 16.6-33.3 General unrestricted release but withGeneral unrestricted release but with warning warningcautioning users of the highcautioning users of the high sampling variablitity. sampling variablitity. Should Should

be identified by letter E.be identified by letter E.

UnacceptableUnacceptable > 33.3> 33.3 No release.No release.

Should be flagged with letter F.Should be flagged with letter F.

Sampling ErrorSampling Error

Measuring sampling error for complex sample designs:Measuring sampling error for complex sample designs:

– Simple formulas not availableSimple formulas not available

– Most software packages do not incorporate design Most software packages do not incorporate design effect (and weights adjustments) appropriately for effect (and weights adjustments) appropriately for calculationscalculations

– Solution for CCHS: the Solution for CCHS: the Bootstrap Bootstrap re-samplingre-sampling methodmethod

Bootstrap methodBootstrap method

Principle:Principle:

– You want to estimate how precise is your estimation of the number of You want to estimate how precise is your estimation of the number of smokers in Canadasmokers in Canada

– You could draw 500 totally new CCHS samples, and compare the 500 You could draw 500 totally new CCHS samples, and compare the 500 estimations you would get from these samples. The variance of these 500 estimations you would get from these samples. The variance of these 500 estimations would indicate the precision.estimations would indicate the precision.

– ProblemProblem: drawing 500 new CCHS samples is $$$: drawing 500 new CCHS samples is $$$

– SolutionSolution: Assuming your sample is representative of the population, sample : Assuming your sample is representative of the population, sample 500 new subsamples and compute new sampling weights for each 500 new subsamples and compute new sampling weights for each subsample.subsample.

Starting point: Full data file (example presented for a given stratum)ID Wgt ClusterA 10 1B 10 1C 10 1D 10 2E 10 2F 10 2G 10 3H 10 3I 10 4J 10 4

Select n-1 clusters among n within each stratum (with replacement)ID Wgt Cluster B1 = # of times the cluster is selectedA 10 1 1B 10 1 1C 10 1 1D 10 2 1E 10 2 1F 10 2 1G 10 3 0H 10 3 0I 10 4 1J 10 4 1

Repeat the process 500 times (*BOOTSTRAP REPLICATES*)ID Wgt Cluster B1 B2 . . . . . . . . . . . . B500A 10 1 1 0 3B 10 1 1 0 3C 10 1 1 0 3D 10 2 1 1 0E 10 2 1 1 0F 10 2 1 1 0G 10 3 0 0 0H 10 3 0 0 0I 10 4 1 2 0J 10 4 1 2 0

Apply the survey weight (Wgt) (*BOOTSTRAP WEIGHTS*)ID Wgt Cluster B1 B2 . . . . . . . . . . . . B500A 10 1 10 0 30B 10 1 10 0 30C 10 1 10 0 30D 10 2 10 10 0E 10 2 10 10 0F 10 2 10 10 0G 10 3 0 0 0H 10 3 0 0 0I 10 4 10 20 0J 10 4 10 20 0

Adjust for the fact that we picked n-1 among n (factor = n / n-1 = 1.33)ID Wgt Cluster B1 B2 . . . . . . . . . . . . B500A 10 1 13 0 40B 10 1 13 0 40C 10 1 13 0 40D 10 2 13 13 0E 10 2 13 13 0F 10 2 13 13 0G 10 3 0 0 0H 10 3 0 0 0I 10 4 13 27 0J 10 4 13 27 0

USING THE BOOTSTRAP WGTS: Estimate the number of smokersID Wgt Cluster Smoke B1 B2 . . . . . . . . . . . . B500A 10 1 X 13 0 40B 10 1 X 13 0 40C 10 1 13 0 40D 10 2 13 13 0E 10 2 13 13 0F 10 2 13 13 0G 10 3 X 0 0 0H 10 3 0 0 0I 10 4 13 26 0J 10 4 X 13 27 0

40 39 27 . . . . . . . . . . . . 80

Bootstrap methodBootstrap method

T = 40Var = (Bi - B)2 / 499

How CCHS Bootstrap weights are createdHow CCHS Bootstrap weights are created(the secret is now revealed!!!)(the secret is now revealed!!!)

Bootstrap MethodBootstrap Method

How Bootstrap replicates are built?How Bootstrap replicates are built? The “real” recipeThe “real” recipe

1- Subsample clusters (SRS) within a design stratum1- Subsample clusters (SRS) within a design stratum

2- Apply (initial design) weight2- Apply (initial design) weight

3- Adjust (boost) weight for selection of n-1 among n3- Adjust (boost) weight for selection of n-1 among n

4- Apply all standard weight adjustments (nonresponse, 4- Apply all standard weight adjustments (nonresponse, integration, share, etc.)integration, share, etc.)

5- Post-stratification to population counts5- Post-stratification to population counts The bootstrap method intends to mimic the same approach The bootstrap method intends to mimic the same approach

used for the sampling and weighting processesused for the sampling and weighting processes


Sampling weightSampling weight versus versus Bootstrap weightsBootstrap weights

– Sampling weight used to compute the Sampling weight used to compute the estimationestimation of a of a

parameter (e.g.: number of smokers)parameter (e.g.: number of smokers)

– Bootstrap weights used to compute the Bootstrap weights used to compute the precisionprecision of the of the

estimation (e.g.: the CV of the number of smokers estimation (e.g.: the CV of the number of smokers

estimation)estimation)


The process of variance estimation is divided into two phases:The process of variance estimation is divided into two phases: Calculation of bootstrap weightsCalculation of bootstrap weights

• Need to be produced only onceNeed to be produced only once

• Done by Statistics Canada methodologistsDone by Statistics Canada methodologists


Variance estimation using bootstrap weightsVariance estimation using bootstrap weights

• Done by anyone - internally or externallyDone by anyone - internally or externally

• Bootstrap weights files distributed with all CCHS files,Bootstrap weights files distributed with all CCHS files,except Public-Use Microdata File (PUMF)except Public-Use Microdata File (PUMF)

– Bootstrap weights are in a separate file (match using Bootstrap weights are in a separate file (match using IDs)IDs)

– Not for PUMF because bootstrap weights reveal Not for PUMF because bootstrap weights reveal confidential infoconfidential info

– PUMF users must proceed through remote access to PUMF users must proceed through remote access to get get ‘ exact ’ variances or use the CV look-up tables‘ exact ’ variances or use the CV look-up tables


Variance estimation using bootstrap weightsVariance estimation using bootstrap weights

• SAS and SPSS (beta) macro programs provided to users SAS and SPSS (beta) macro programs provided to users (BOOTVAR)(BOOTVAR)

• Allow users to perform a few statistical analysis (totals, Allow users to perform a few statistical analysis (totals, proportions, differences of proportions and regression analysis)proportions, differences of proportions and regression analysis)

• Fully documented with examplesFully documented with examples

• Bootstrap hands-on workshop Bootstrap hands-on workshop

How to use the Bootvar programHow to use the Bootvar program

STEP #1STEP #1

Create your ‘‘analytical file”Create your ‘‘analytical file”

Read CCHS data fileRead CCHS data file

Prepare the necessary dummy Prepare the necessary dummy

variablesvariables

Keep only the necessary Keep only the necessary

variablesvariables

Perform the analysis to obtain Perform the analysis to obtain

the point estimatesthe point estimates

(not essential but recommended)(not essential but recommended)

STEP #2STEP #2

Compute your variances using Compute your variances using

BootvarBootvar

Specify the location of the filesSpecify the location of the files:: Your “analytical file”Your “analytical file” Bootstrap weights fileBootstrap weights file

Specify the level of geographySpecify the level of geography

Specify the analysis to performSpecify the analysis to perform Total, proportion, diff. of prop.Total, proportion, diff. of prop. Regression (linear & logistic)Regression (linear & logistic) Generalized linear modelGeneralized linear model

How to use the Bootvar programHow to use the Bootvar program

Statistical analysisStatistical analysis Using the NPHS cycle 3 (1998) cross-sectional Using the NPHS cycle 3 (1998) cross-sectional dummydummy data, estimate the number of ontarians data, estimate the number of ontarians

aged 12, by gender, who perceive themselves as being:aged 12, by gender, who perceive themselves as being:

- in - in poorpoor or or fairfair health, health,

- in - in goodgood health, health,

- in - in very goodvery good health, health,

- in - in excellentexcellent health. health.

- Compute 95% confidence interval for each point estimate using the Bootvar program.- Compute 95% confidence interval for each point estimate using the Bootvar program.

Necessary variables for the analysisNecessary variables for the analysis

Self-perceived health Self-perceived health (GHC8DHDI)(GHC8DHDI)

0 = poor, 1 = fair, 2 = good, 3 = very good, 4 = excellent, 9 = not stated0 = poor, 1 = fair, 2 = good, 3 = very good, 4 = excellent, 9 = not stated

AgeAge (DHC8_AGE) (DHC8_AGE) SexSex (DHC8_SEX) (DHC8_SEX)

>= 12>= 12 1 = male, 2 = female 1 = male, 2 = female

ProvinceProvince (PRC8_CUR) (PRC8_CUR) Sampling weightSampling weight (WT68) (WT68)

35 = Ontario35 = Ontario

Record identifier for the householdRecord identifier for the household (REALUKEY) (REALUKEY)

Number identifying the person in the householdNumber identifying the person in the household (PERSONID) (PERSONID)

Basic theoritical notions for estimating Basic theoritical notions for estimating a proportiona proportion

Example of a data fileExample of a data file

IDID WeightWeight SexSex Asthma Asthma Asthma_idAsthma_id

AA 50 50 M M YES YES 1 1

BB 60 60 M M NO NO 0 0

CC 50 50 M M NO NO 0 0

DD 70 70 M M YES YES 1 1

E E 50 50 M NO 0 M NO 0

((WeightWeightAA + Weight + WeightDD))

(Weight(WeightAA+Weight+WeightBB+Weight+WeightCC+Weight+WeightDD+Weight+WeightEE))

= (50 + 70) = (50 + 70) // (50 + 60 +50 + 70 + 50) * 100 = 120 (50 + 60 +50 + 70 + 50) * 100 = 120 / 280 * 100 = 43% / 280 * 100 = 43%

%(asthma_men) = __________________________________________ * 100

Little trick for the statistical analysisLittle trick for the statistical analysis

Create your univariate dummy variable Create your univariate dummy variable : :

Men = 1,0Men = 1,0 ((menmen)) Good health = 1,0Good health = 1,0 ((goodgood) ) Men in good health : Men in good health : mgood = men * goodmgood = men * good

men * goodmen * good = mgood= mgood

11 00 00

11 11 11

00 00 00

00 11 00

Results of the statistical analysisResults of the statistical analysis Self-perceived health of ontarians aged 12 or older by gender Self-perceived health of ontarians aged 12 or older by gender

in 1998in 1998

# (‘000) # (‘000) 95% CI 95% CI %% 95% CI 95% CI

MenMen - Poor - Poor / fair/ fair 391391 (330 ; 452)(330 ; 452) 8.48.4 (7.1 ; 9 .8)(7.1 ; 9 .8)

- Good- Good 1,1061,106 (1,007 ; 1,204)(1,007 ; 1,204) 23.923.9 (21.7 ; 26.0)(21.7 ; 26.0)

- Very good- Very good 1,7641,764 (1,648 ; 1,880)(1,648 ; 1,880) 38.138.1 (35.6 ; 40.6)(35.6 ; 40.6)

- Excellent- Excellent 1,3731,373 (1,268 ; 1,479)(1,268 ; 1,479) 29.629.6 (27.4 ; 31.9)(27.4 ; 31.9)

WomenWomen - Poor / fair- Poor / fair 480480 (409 ; 551)(409 ; 551) 9.99.9 (8.5 ; 11.4)(8.5 ; 11.4)

- Good- Good 1,2581,258 (1,151 ; 1,364)(1,151 ; 1,364) 26.126.1 (23.9 ; 28.3)(23.9 ; 28.3)

- Very good- Very good 1,8461,846 (1,726 ; 1,965)(1,726 ; 1,965) 38.238.2 (35.8 ; 40.7)(35.8 ; 40.7)

- Excellent- Excellent 1,2431,243 (1,138 ; 1,348)(1,138 ; 1,348) 25.825.8 (23.6 ; 27.9)(23.6 ; 27.9)

Why use the Bootstrap methodWhy use the Bootstrap method??

Other techniques:Other techniques:

TaylorTaylor

– Need to define a linear equation for each statistic Need to define a linear equation for each statistic examinedexamined

JacknifeJacknife

– Number of replicates depends on the number of Number of replicates depends on the number of strata (large number of strata makes it impossible to strata (large number of strata makes it impossible to disseminate)disseminate)

Why use the Bootstrap methodWhy use the Bootstrap method??

BOOTSTRAPBOOTSTRAP

more user-friendly when there is a large number of strata more user-friendly when there is a large number of strata sets of 500 bootstrap weights can be distributed to data userssets of 500 bootstrap weights can be distributed to data users Recommended (over the jackknife) for estimating the variance Recommended (over the jackknife) for estimating the variance

of nonsmooth functions like quantiles, LICOof nonsmooth functions like quantiles, LICO

Official reference:Official reference:

– ““Bootstrap Variance Estimation for the National Population Bootstrap Variance Estimation for the National Population Health Survey”, Health Survey”, D. Yeo, H. Mantel, and T.-P. Liu. 1999, D. Yeo, H. Mantel, and T.-P. Liu. 1999, Baltimore, ASA Conference.Baltimore, ASA Conference.

CV Look-up TablesCV Look-up Tables

Alternative to bootstrapAlternative to bootstrap ApproximateApproximate Can only be used for categorical variables, and for Can only be used for categorical variables, and for

estimations of totals and proportionsestimations of totals and proportions Available for every health region, province and CanadaAvailable for every health region, province and Canada Provided with PUMF and Provided with PUMF and Share fileShare file for some for some

subpopulationssubpopulations

CV Look-up Tables—ExampleCV Look-up Tables—Example

National Population Health Survey ‑ 1996/1997

Approximate Sampling Variability Tables for Ontario Health Area:OTTAWA CARLETON ‑ Selected members

NUMERATOR OF ESTIMATED PERCENTAGE PERCENTAGE ('000) 0.1% 1.0% 2.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 50.0% 70.0% 90.0%

1 ******** 48.6 48.4 47.6 46.4 45.0 43.7 42.3 40.9 39.4 37.8 34.5 26.8 15.5 2 ******** 34.4 34.2 33.7 32.8 31.9 30.9 29.9 28.9 27.9 26.8 24.4 18.9 10.9 3 ******** 28.1 27.9 27.5 26.8 26.0 25.2 24.4 23.6 22.7 21.9 19.9 15.5 8.9 4 ******** 24.3 24.2 23.8 23.2 22.5 21.9 21.2 20.4 19.7 18.9 17.3 13.4 7.7 5 ******** 21.7 21.6 21.3 20.7 20.1 19.5 18.9 18.3 17.6 16.9 15.5 12.0 6.9 6 ******** 19.8 19.7 19.4 18.9 18.4 17.8 17.3 16.7 16.1 15.5 14.1 10.9 6.3 7 ******** 18.4 18.3 18.0 17.5 17.0 16.5 16.0 15.5 14.9 14.3 13.1 10.1 5.8 8 **************** 17.1 16.8 16.4 15.9 15.5 15.0 14.5 13.9 13.4 12.2 9.5 5.5 9 **************** 16.1 15.9 15.5 15.0 14.6 14.1 13.6 13.1 12.6 11.5 8.9 5.2 10 **************** 15.3 15.1 14.7 14.2 13.8 13.4 12.9 12.5 12.0 10.9 8.5 4.9 ... ... 300 **************************************************************************************** 2.0 1.5 0.9 350 **************************************************************************************** 1.8 1.4 0.8 400 ************************************************************************************************ 1.3 0.8 450 ************************************************************************************************ 1.3 0.7 500 ************************************************************************************************ 1.2 0.7

NOTE: FOR CORRECT USAGE OF THESE TABLES PLEASE REFER TO MICRODATA DOCUMENTATION

Another example using the Bootvar programAnother example using the Bootvar program

Statistical analysisStatistical analysis

Using the NPHS cycle 3 (1998) cross-sectional Using the NPHS cycle 3 (1998) cross-sectional dummydummy data, determine data, determine whether or not thewhether or not the number of men number of men aged 12 or older who perceive aged 12 or older who perceive themselves as being in themselves as being in excellent healthexcellent health in Ontario is statistically different in Ontario is statistically different (at level (at level =5%) than the =5%) than the number of women.number of women.

Basic theoritical notions for performing Basic theoritical notions for performing a Z-testa Z-test

M_excel = estimated proportion of men in excellent healthM_excel = estimated proportion of men in excellent health

F_excel = estimated proportion of women in excellent healthF_excel = estimated proportion of women in excellent health

Hypothesis testHypothesis test:: HH00: M_excel = F_excel: M_excel = F_excel

HH11: M_excel : M_excel F_excel F_excel

At level At level = 0,05, we conclude H = 0,05, we conclude H00 if if | z | <= 1.96| z | <= 1.96

We conclude HWe conclude H11 otherwise. otherwise.

Z = ( M_excel - F_excel ) Z = ( M_excel - F_excel )

sd (M_excel-F_excel) sd (M_excel-F_excel)

We use the section “difference of proportions” of the BOOTVAR program to estimate the standard deviation of the difference between the two We use the section “difference of proportions” of the BOOTVAR program to estimate the standard deviation of the difference between the two estimates.estimates.

__________________

ResultsResults

M_excel = 29.64% ; F_excel = 25.75% ; sd(M_excel-F_excel) = 1.62M_excel = 29.64% ; F_excel = 25.75% ; sd(M_excel-F_excel) = 1.62

Z = ( M_excel - F_excel ) Z = ( M_excel - F_excel ) = (29.64 - 25.75) = 3.89 = = (29.64 - 25.75) = 3.89 = 2.402.40

sd (M_excel-F_excel) sd (M_excel-F_excel) 1.62 1.62 1.62 1.62

At At = 0,05 level , we conclude H = 0,05 level , we conclude H11 because because z = 2.40 > 1.96 .z = 2.40 > 1.96 .

We can then conclude that among the ontarians aged 12 or older there is a statistical difference between men and We can then conclude that among the ontarians aged 12 or older there is a statistical difference between men and women with regard to the caracteristic women with regard to the caracteristic “self-perceived health = excellent”.“self-perceived health = excellent”.

________________ _________ ____

CCHS - Data Dissemination StrategyCCHS - Data Dissemination Strategy

Wide range of users and capacity–136 health regions

–13 provincial/territorial Ministries of Health

–Health Canada and CIHI

–Internal STC analysts

–Academics

–Others

Data products–Microdata

–Analytical products (Health Reports, How Healthy are Canadians, etc…)

–Tabular statistics (ePubs, Cansim II, community profiles, etc…)

–Client support (head and regional offices, CCHS website, workshops, etc…)

CCHS - Access to microdataCCHS - Access to microdata

Master file–all records, all variables

•Statistics Canada

•university research data centres

•remote access

Share / Link files–respondents who agreed to share / link

•provincial/territorial Ministries of Health

•health regions (through the STC third-party share agreement)

Public Use Microdata File (PUMF)–all records, subset of variables with collapsed response categories

•free for 136 health regions

•cost recovery for others

CCHS - Overview of Cycle 1.2CCHS - Overview of Cycle 1.2

Produce provincial cross-sectional estimates from a sample of 30,000 respondents

Area frame sample only / one person per household

CAPI only

90 minute in-depth interviews on mental health and well-being based on WMH2000 questionnaire

Scheduled to begin collection in May 2002

CCHS - Future PlansCCHS - Future Plans

Same two-year cycle approach:–health region level survey starting in January 2003

–provincial level survey starting in January 2004

New consultation process with provincial and regional authorities

Flexible sample designs (adaptable to regional needs)

Development of an in-depth nutrition focus content (Cycle 2.2)

CCHS Web siteCCHS Web site

www.statcan.ca/health_surveyswww.statcan.ca/health_surveys

www.statcan.ca/enquetes_santéwww.statcan.ca/enquetes_santé

Contacts in MethodologyContacts in Methodology

Yves Béland:Yves Béland:[email protected]@statcan.ca

François Brisebois: François Brisebois: [email protected]@statcan.ca

Documents

Canadian Community Health Survey Cycle 1.1 Overview of methodological issues and more