Embed Size (px)
Citation preview
4/24/2012
1
The Effects of Raking and Cell Phone Integration on BRFSS Outcomes
Machell Town, M.S.
Carol Pierannunzi, Ph.D.
Division of Behavioral Surveillance
Office of Surveillance, Epidemiology, and Laboratory Services
Division of Behavioral Surveillance
4/24/2012
2
Brief Agenda
Weighting procedures Design weights
Post stratification
Iterative proportional fitting
Why change weighting procedures now? Cell phone
Computer capacity
Impact of changes on estimation BRFSS
Examples of small and large impact
Changes when cell phones are incorporated
Conclusions
Brief look at state level phone use data (preliminary)
4/24/2012
3
WEIGHTING PROCEDURES
4/24/2012
4
Design and GeoStrata Weighting
Takes into account the geographic region/strata of the sample.
Design weight uses number of adults in household and number of phones in household for landline sample.
BRFSS landline sample is drawn using low/high density strata within each of the regions (usually smaller than states)
Stratum weight (_STRWT) = NRECSTR/ NRECSEL
4/24/2012
5
Calculating the Design Weight
Design Weight = _STRWT* (1/NUMPHON2) * NUMADULT NUMPHON2= number of phones within the household
NUMADULT = number of adults eligible for the survey within the household
Questions for the design weights are asked in screening questions and in demographic sections of the survey
4/24/2012
6
Data Weighting
Data weights take the design weighting and incorporate characteristics of the population and the sample
Final Weights (_FINALWT) = Design Weight * some form of data weighting In past BRFSS used post stratification
In 2008 Iterative Proportional Fitting was first used
In 2011 Iterative Proportional Fitting will be only method of data weighting for BRFSS
4/24/2012
7
WEIGHTING Post -Stratification
4/24/2012
8
Where We Have Been--- Post Stratification
Post Stratification is based on known demographics of the population. For BRFSS Post stratification included:
· Regions within states
· Race/ Ethnicity (in detailed categories)
· Gender
· Age (in 7 categories)
Post-stratification forces the sum of the weighted frequencies to equal the population estimates for the region or state by race, age ,and gender.
Post stratification weights are applied to the responses, allowing for estimates of how groups of non-respondents would have answered survey questions.
4/24/2012
9
Post-stratification
Post-stratification Adjustment Factor is calculated for each race/ethnicity, gender, and age group combination. Requires knowledge of each subset of each factor at the
geographic level of interest –otherwise categories must be collapsed
Requires a minimum number of persons in each cell—otherwise categories must be collapsed
_POSTSTR = Population/Design weight within the weighting class cell.
4/24/2012
10
Weight Trimming
Sometimes post-stratification resulted in very small or disproportionately large weights within age/race/gender/region categories.
Weight trimming or category collapsing would be done if categories were disproportionately large or too small (< 50 responses).
4/24/2012
11
WEIGHTING Iterative Proportional Fitting (Raking)
4/24/2012
12
Iterative Proportional Fitting
Rather than adjusting weights to categories, IPF adjusts for each dimension separately in an iterative process. The process will continue up to 75 times, or until data converges to Census estimates.
Region
Age
Race
Gender
Phone Type
Home Ownership
Education
Marital Status
Gender by Race
Age by Gender
Age by Race
4/24/2012
13
New Variables Introduced as Controls With IPF
Education
Marital status
Home ownership/renter
Telephone source (cell phone or landline)
4/24/2012
14
Post Stratification vs. Iterative Proportional Fitting
Post Stratification
Iterative Proportional
Fitting
Operates with less computer time
Allows for incorporation of new variables. Allows for incorporation of cell phone data. Seems to more accurately represent population data (reduces bias).
4/24/2012
15
Why Incorporate IPF Now?
Computer capacity has increased.
Cell phones are becoming larger percentage of the total number of calls.
Noncoverage with declining response rates makes weighting more important than ever.
4/24/2012
16
Examples of IPF From 2010 Data
Note that example may be slightly different from 2011 analyses because We did not collect home ownership at that time
We still used phone interruption variable
Some of the iterations are different than will be conducted on 2011 dataset
4/24/2012
17
Raking – Iteration 1
17
First Control Variable
Output
Weight
Sum of
Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
Age 18-24,Male 87122.60 95468 -8345.40 6.533 7.159 -0.626
Age 18-24,Female 77180.40 90249 -13068.60 5.788 6.768 -0.980
Age 25-34,Male 109419.36 118670 -9250.64 8.206 8.899 -0.694
Age 25-34,Female 114395.17 112007 2388.17 8.579 8.400 0.179
Age 35-44,Male 121328.71 117184 4144.71 9.099 8.788 0.311
Age 35-44,Female 115609.98 113779 1830.98 8.670 8.533 0.137
Age 45-54,Male 138658.26 127077 11581.26 10.398 9.530 0.869
Age 45-54,Female 136904.33 127439 9465.33 10.267 9.557 0.710
Age 55-64,Male 90338.77 95032 -4693.23 6.775 7.127 -0.352
Age 55-64,Female 91693.43 97422 -5728.57 6.876 7.306 -0.430
Age 65-74,Male 57475.54 54171 3304.54 4.310 4.062 0.248
Age 65-74,Female 62709.50 61828 881.50 4.703 4.637 0.066
Age 75+,Male 49772.58 46515 3257.58 3.733 3.488 0.244
Age 75+,Female 80867.37 76635 4232.37 6.064 5.747 0.317
Should be
│.025│ or less
4/24/2012
18
Raking – Iteration 1
18
Second Control Variable
Output
Weight Sum
of Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target %
of
Weights
Difference
in %
WH NH 1151321.16 1156947 -5625.84 86.340 86.762 -0.422
OT NH 15305.42 12036 3269.42 1.148 0.903 0.245
HISP 85300.51 84230 1070.51 6.397 6.317 0.080
BL NH,AS NH,AI NH 81548.91 80263 1285.91 6.116 6.019 0.096
4/24/2012
19
Raking - Iteration 1
Third Control Variable
Input
Weight Sum
of Weights
Target
Total
Sum of
Weights
Difference
% of
Input
Weights
Target %
of Weights
Difference
in %
Less than HS 89962.05 143928 -53966.35 6.746 10.793 -4.047
HS Grad 412857.99 414505 -1646.81 30.961 31.085 -0.123
Some College 388163.96 448218 -60054.20 29.109 33.613 -4.504
College Grad 442492.00 326825 115667.37 33.183 24.509 8.674
19
4/24/2012
20
Raking – Iteration 1
20
Fourth Control Variable
Output
Weight
Sum of
Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
Married 816399.38 792326 24073.29 61.223 59.418 1.805
Never married, member
unmarried couple
277180.73 300111 -22930.01 20.786 22.506 -1.720
Divorced, Widowed, Separated 239895.88 241039 -1143.29 17.990 18.076 -0.086
Fifth Control Variable
Output
Weight Sum
of Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
Phone interruption 78558.62 82944 -4385.49 5.891 6.220 -0.329
No Phone Interruption 1254917.38 1250532 4385.49 94.109 93.780 0.329
4/24/2012
21
Raking – Iteration 1
21
Sixth Control Variable
Output
Weight
Sum of
Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
Male, WH NH 553107.34 552171 936.34 41.479 41.408 0.070
Male, BL NH,AS NH,AI NH,OT
NH,HISP
101008.49 101946 -937.51 7.575 7.645 -0.070
Female, WH NH 598213.82 604776 -6562.18 44.861 45.353 -0.492
Female, HISP 38304.69 32837 5467.69 2.873 2.463 0.410
Female, BL NH,AS NH,AI NH,OT
NH
42841.66 41746 1095.66 3.213 3.131 0.082
4/24/2012
22
Raking – Iteration 1
22
Seventh Control Variable
Output
Weight
Sum of
Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
18-34, WH NH 308020.95 332809 -24788.05 23.099 24.958 -1.859
18-34, BL NH,AS NH,AI NH,OT
NH,HISP
80096.58 83585 -3488.42 6.007 6.268 -0.262
35-54, WH NH 442299.71 421539 20760.71 33.169 31.612 1.557
35-54, BL NH,AS NH,AI NH,OT
NH,HISP
70201.57 63940 6261.57 5.265 4.795 0.470
55+, WH NH 401000.50 402599 -1598.50 30.072 30.192 -0.120
55+, BL NH,AS NH,AI NH,OT
NH,HISP
31856.70 29004 2852.70 2.389 2.175 0.214
4/24/2012
23
Raking – Iteration 1
23
Eighth Control Variable
Output
Weight
Sum of
Weights
Target
Total
Sum of
Weights
Difference
% of
Output
Weights
Target % of
Weights
Difference
in %
Cell Phone Only 210390.11 197088 13302.35 15.778 14.780 0.998
Landline Only 270206.34 280297 -10090.31 20.263 21.020 -0.757
Landline and Cell Phone 852879.55 856092 -3212.04 63.959 64.200 -0.241
4/24/2012
24
Raking – Iteration 2
24
First Control Variable
Output
Weight
Sum of
Weights
Target
Total
% of
Output
Weights
Target %
of Weights
Difference
in % from
Iteration1
Difference
in %
Age 18-24,Male 94727.80 95468 7.104 7.159 -0.626 -0.056
Age 18-24,Female 87222.36 90249 6.541 6.768 -0.980 -0.227
Age 25-34,Male 116312.81 118670 8.723 8.899 -0.694 -0.177
Age 25-34,Female 110348.83 112007 8.275 8.400 0.179 -0.124
Age 35-44,Male 118670.65 117184 8.899 8.788 0.311 0.111
Age 35-44,Female 113723.15 113779 8.528 8.533 0.137 -0.004
Age 45-54,Male 130207.90 127077 9.765 9.530 0.869 0.235
Age 45-54,Female 130419.01 127439 9.780 9.557 0.710 0.223
Age 55-64,Male 93001.49 95032 6.974 7.127 -0.352 -0.152
Age 55-64,Female 96092.37 97422 7.206 7.306 -0.430 -0.100
Age 65-74,Male 54156.67 54171 4.061 4.062 0.248 -0.001
Age 65-74,Female 62303.45 61828 4.672 4.637 0.066 0.036
Age 75+,Male 47039.67 46515 3.528 3.488 0.244 0.039
Age 75+,Female 79249.83 76635 5.943 5.747 0.317 0.196
4/24/2012
25
Raking - Iteration 7
First Control Variable
Output
Weight
Sum of
Weights
Target
Total
% of
Output
Weights
Target
% of
Weights
Difference
in % from
Iteration1
Difference
in %
Age 18-24,Male 95491.87 95468 7.161 7.159 -0.626 0.002
Age 18-24,Female 90265.83 90249 6.769 6.768 -0.980 0.001
Age 25-34,Male 118621.93 118670 8.896 8.899 -0.694 -0.004
Age 25-34,Female 111985.21 112007 8.398 8.400 0.179 -0.002
Age 35-44,Male 117205.13 117184 8.789 8.788 0.311 0.002
Age 35-44,Female 113769.71 113779 8.532 8.533 0.137 -0.001
Age 45-54,Male 127088.93 127077 9.531 9.530 0.869 0.001
Age 45-54,Female 127437.46 127439 9.557 9.557 0.710 -0.000
Age 55-64,Male 95037.18 95032 7.127 7.127 -0.352 0.000
Age 55-64,Female 97426.08 97422 7.306 7.306 -0.430 0.000
Age 65-74,Male 54168.73 54171 4.062 4.062 0.248 -0.000
Age 65-74,Female 61831.76 61828 4.637 4.637 0.066 0.000
Age 75+,Male 46503.23 46515 3.487 3.488 0.244 -0.001
Age 75+,Female 76642.96 76635 5.748 5.747 0.317 0.001
25
All less than
│.025│
4/24/2012
26
Raking - Iteration 7
Eighth Control
Variable
Output
Weight Sum
of Weights
Target
Total
% of
Output
Weights
Target %
of
Weights
Difference
in % at
Iteration
1
Difference
in %
Cell Phone Only 197101.32 197088 14.781 14.780 0.998 0.001
Landline Only 280285.25 280297 21.019 21.020 -0.757 -0.001
Landline and
Cell Phone
856089.43 856092 64.200 64.200 -0.241 -0.000
26
**** Program terminated at iteration 7 because all
current percents differ from target percents by less
than 0.025*****
4/24/2012
27
IMPACT OF CHANGING TO RAKING (IPV) ON THE BRFSS
4/24/2012
28
BRFSS 2010 Combined States a Data Difference In Weighted Percentages
A Excludes AK, DC, TN, SD
0
10
20
30
40
50
60
70
80
LL Post stratified LL Raking LLCP Raking
4/24/2012
29
Marginal Changes Weighted Percentages for Demographic Characteristics, BRFSS 2010
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
4/24/2012
30
BRFSS 2010 Combined States Data
Difference In Weighted Percentages of Health Outcomes
0
5
10
15
20
25
30
35
LL Post stratified LL Raking LLCP Raking
A Excludes AK, DC, TN, SD
4/24/2012
31
Marginal Changes for in Weighted Percentage s Health Outcomes, BRFSS 2010
0
0.2
0.4
0.6
0.8
1
1.2
4/24/2012
32
Weighted Prevalence Estimates for Current Smoker by Year, Weighting Method
0
5
10
15
20
25
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Pre
va
len
ce E
stim
ate
Year
Landline Post Stratification Landline Raking Weighting
Landline/ Cell Phone Raking Weighting
NOTE: All US states and territories except SD and TN
4/24/2012
33
STATE LEVEL OUTCOMES
4/24/2012
34
In Some Cases, Small Changes (Landline Only)
Table 1
State-level Responses to Question:
―Has a doctor, nurse or other healthcare provider ever told you that you have diabetes?‖
By Type Of Weighting Procedure for Landline Data
Response Landline
Weighted
frequency with
Post-
Stratification
Landline Percent
With Post-
Stratification
Landline
Weighted
frequency
with Raking
Landline
Percent
With
Raking
Differences in
Landline
Percentages
(Post-
Stratification-
Raking)
Yes 434,858 12.26 440,694 12.43 -0.17
Yes, but only
during pregnancy 26,306 0.74 26,262 0.74 0.00
No 3,031,681 85.44 3,029,545 85.42 0.02
No, Pre-diabetes/
borderline
diabetes
55,454 1.56 50,196 1.42 0.15
4/24/2012
35
In Some Cases, Larger Differences– But Not Consistent Differences
(Landline Only)
Table 2
State-level Responses to Question:
―Would you say that in general your health is excellent, very good, good, fair or poor?‖
By Type Of Weighting Procedure for Landline Data
Response Landline Weighted
Frequency With Post-
Stratification
Landline
Percent
With Post-
Stratification
Landline
Weighted
Frequency
With Raking
Landline
Percent
With
Raking
Differences In
Landline
Percentages
(Post-Stratification
- Raking)
Excellent 631,742 17.83 575,541 16.27 1.56
Very Good 1,037,345 29.27 963,330 27.23 2.04
Good 1,107,272 31.26 1,111,484 31.42 -0.16
Fair 519,248 14.65 591,716 16.73 -2.07
Poor 247,424 6.98 295,425 8.35 -1.37
4/24/2012
36
In Some Cases, Consistent Differences (Landline Only)
Table 3
State-level Responses to Question:
―During the past month, other than your regular job, did you participate in any physical activities or
exercises such as running, calisthenics, golf, gardening, or walking for exercise?‖
By Type Of Weighting Procedure for Landline Data
Response Landline Weighted
Frequency With Post-
Stratification
Landline
Percent
With Post-
Stratification
Landline
Weighted
Frequency
With
Raking
Landline
Percent
With
Raking
Differences In
Landline
Percentages
(Post-Stratification
- Raking)
Yes 2,448,288 68.97 2,342,381 65.98 2.99
No 1,101,378 31.03 1,207,643 34.02 -2.99
4/24/2012
37
But Differences Go Away Sometimes When Cell Phones Are Added
Table 4
State-level Responses to Question:
―During the past month, other than your regular job, did you participate in any physical activities or exercises
such as running, calisthenics, golf, gardening, or walking for exercise?‖
By Type Of Weighting Procedure for Landline and Cell Phone Data
Response Landline
Weighted
Frequency
With Post-
Stratification
Landline
Percent
With Post-
Stratification
Landline
Weighted
Frequency
With Raking
Landline
Percent
With
Raking
Differences In
Landline
Percentages
(Post-
Stratification -
Raking)
Landline
And Cell
Phone
Weighted
Frequency
With Raking
Landline
And Cell
Phone
Percent
Landline And
Cell Phone
Differences In
Percentages
(Post-
Stratification -
Raking)
Yes 2,448,288 68.97 2,342,381 65.98 2.99 2,447,823 68.96 0.02
No 1,101,378 31.03 1,207,643 34.02 -2.99 1,102,053 31.04 -0.02
4/24/2012
38
Persistent Differences May Exist Even When Adding Cell Phone Responses
Table 5
State-level Responses to Question:
―Do you smoke cigarettes every day, some days or not at all?‖
By Type Of Weighting Procedure for Landline and Cell Phone Data
Response Landline
Weighted
Frequency
With Post-
Stratification
Landline
Percent
With Post-
Stratification
Landline
Weighted
Frequency
With
Raking
Landline
Percent
With
Raking
Differences In
Landline
Percentages
(Post-
Stratification -
Raking)
Landline
And Cell
Phone
Weighted
Frequency
With
Raking
Landline
And Cell
Phone
Percent
Landline And
Cell Phone
Differences In
Percentages
(Post-
Stratification -
Raking)
Every
day
581,967 36.32 704,831 40.95 -4.63 676,129 40.40 -4.08
Some
Days
213,724 13.34 248,782 14.45 -1.12 199,278 11.91 1.43
Not At
All
806,827 50.35 767,708 44.60 5.75 798,181 47.69 2.65
4/24/2012
39
CONCLUSIONS
4/24/2012
40
Conclusions (1)
New weighting procedures are needed to keep pace with changes in personal communications.
The inclusion of new variables and more complex weighting procedures for large scale survey data are now feasible, because of improvements in computer capacity.
There will be some differences in estimates when weighting procedures change and when new variables for weighting are introduced.
Examples shown here are only depictions of potential outcomes of changes at the BRFSS.
4/24/2012
41
Conclusions (2)
Good news: demographic characteristics adjusted to more closely match Census data.
Most health outcomes indicate increases in risk behaviors (especially when state data are aggregated).
Some increases in chronic conditions, but uneven across states.
4/24/2012
42
Thank You
For more information please contact Centers for Disease Control and Prevention
1600 Clifton Road NE, Atlanta, GA 30333
Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348
E-mail: [email protected] Web: http://www.cdc.gov
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Office of Surveillance, Epidemiology, and Laboratory Services
Division of Behavioral Surveillance
