Validation in Statistics Canada Health Surveys

Presentation to RRFSS Workshop

June 20, 2007Vincent Dale

Outline

Statistics Canada quality assurance framework

Ensuring data accuracy Past validation projects Future projects Future directions

Quality Assurance Framework

Relevance Degree to which information meets the real needs of data users

Timeliness Delay between the reference period and the date upon which the information becomes available

Accessibility Ease with which information can be obtained from the Agency

Accuracy Degree to which the information correctly describes the phenomena it was designed to measure

Interpretability Availability of supplementary information and metadata necessary to interpret and utilize information appropriately

Coherence Degree to which information can be successfully brought together with other statistical information within a broad analytic framework and over time

Statistics CanadaQuality Assurance Framework

Trade-offs between aspects of quality These are actively managed through a

variety of processes, including:– User and stakeholder feedback mechanisms– Program review– Data analysis and dissemination– Standards and documentation (concepts,

variables, classifications)

Ensuring Data Accuracy

Questionnaire development– Wherever possible, validated questionnaire modules are

used Sometimes modified for use in population-based survey Sometimes not as valid as advertised

Questionnaire testing– STC policy requires testing of all new questionnaires

Cognitive interviews and focus groups

Coherence versus accuracy– Sometimes better to keep measure stable even if imperfect

Ensuring Data Accuracy

Sampling error – error attributed to studying a fraction of a population rather

than carrying out a census

Non-sampling error – coverage errors– response errors– non-response errors– processing errors– estimation errors– analysis errors

Ensuring Data Accuracy

Explosion of health survey data– More data, more often for smaller levels of

geography

Increasing attention paid to validity– Health measures– Administrative data– Complimentary surveys

What is validity?

Face validity Internal validity

– construct validity

External validity– Criterion– Sensitivity, specificity, predictive value

Past CCHS Validation Projects

Health Care Utilisation

– Data linkage of CCHS responses with BC administrative health records

– Supplemented with analysis of: Respondent interpretation and formulation of responses Interviewer behaviour and training Patterns in response changes, edits and timing of

response entry

Contacts with Health Professionals

Results of linkage:– Compared to provincial health records:

Most CCHS respondents (58%) reported fewer primary care physician contacts

On average, CCHS respondents reported 1.7 fewer primary care physician contacts

Older CCHS respondents and respondents with better self-perceived health tended to report fewer contacts

Younger respondents and respondents with poorer self-perceived health tended to report more contacts

Contacts with Health Professionals

Recommendations from study:– Revise wording of specific questions to minimize

misinterpretation– Facilitate consistent interviewer probing

techniques– Improved edits and CAPI/CATI application

navigation for interviewers to facilitate changes to previously-answered questions

Evaluation of coverage of linked CCHS and hospital inpatient records

Probabilistic linkage used to identify CCHS 1.1 respondents (excluding Québec) hospitalized over a 14-month period– Health person-oriented information database

(HPOI) is a virtual census of hospital admissions and used as the standard

Survey weights applied to the 8230 CCHS records which were found in the HPOI database

Evaluation of coverage of linked CCHS and hospital inpatient records

Number hospitalized in acute-care hospitals, Sept. 1, 2000 – Nov. 3, 2001, aged 12+, Canada excluding Québec

CCHS HPOI

Unweighted

(n)

Weighted

(N)

Count

(N)

Coverage rate

TOTAL 8,230 1,334,909 1,612,269 82.8%

Evaluation of coverage of linked CCHS and hospital inpatient records

Under-reporting rates similar between women and men

– Lower among Manitoba residents (69.2%)– Higher among individuals aged 12-74 (86.1%) than those

aged 75+ (70.3%)

Under-reporting is an essential prerequisite to further analyses based on the CCHS – HPOI linked data

– Use of the linked file could lead to bias due depending on province/territory of residence and age

CCHS Measured Height & Weight

In 2005, height / weight were measured for a sub sample of CCHS Cycle 3.1 participants (n=4567)– Weight: mean difference between measured and

self-reported weight of 2.1 kg (2.5 kg for women)– Height: mean difference between measured and

self-reported height of -0.7 cm (-1.0 cm for men)– BMI: mean difference between measured and

self-reported BMI was 1.1

CCHS Measured Height & Weight

Overweight Obese Class II, III

(25.0 to 29.9)

Total % Total % Total % Total % Total %

('000) ('000) ('000) ('000) ('000)

Self-reported BMI category

Both sexes

Underweight (<18.5) 271 67 308 3 1 0 1 0 0 0Normal weight (18.5 to 24.9) 131 33 10,163 94 2,651 30 120 3 4 0Overweight (25.0 to 29.9) 0 0 388 4 5,851 67 1,894 44 134 9Obese Class I (30.0 to 34.9) 0 0 0 0 244 3 2,247 52 603 39Obese Class II, III (=35) 0 0 0 0 0 0 22 1 822 53

Total 402 100 10,859 100 8,746 100 4,288 100 1,562 100

(<18.5) (18.5 to 24.9) (30.0 to 34.9) (=35)

Measured BMI category

Underweight Normal weight Obese Class I

CCHS Mode Effect Study

Potential differences associated with two methods of collection used in CCHS

– CAPI: computer assisted personal interview– CATI: computer assisted telephone interview

Used a split-panel design with a unique sample frame

– secondary sampling units randomly assigned to CAPI or CATI.

– Fully integrated as part of CCHS cycle 2.1– 11 sites selected to provide a good representation of each

region in Canada

CCHS Mode Effect Study

Important differences observed for obesity rates – CAPI = 17.9%; CATI = 13.2%

Physical activity index – inactive persons– CAPI = 42.3%; CATI = 34.4%

Statistically significant differences for contact with medical doctors and unmet health care needs

No significant differences observed in the vast majority of health indicators

CCHS Mode Effect Study

Overall results show that cycles 1.1 and 2.1 are largely comparable despite an increase in CATI collection for Cycle 2.1 (2003)

Results led to a decision to measure exact height and weight for a sub-sample of respondents in cycle 3.1 (2005)

Led to improved standardization of interviewer procedures across the two collection modes

Future Validation Projects

Scale Reliability - Factor Analysis

Construct validity / scale reliability:– Cronbach’s Alpha calculated for scales used in

CCHS questionnaire– Results could be published in user guide

What are standards? Some researchers feel that scores should be above 0.8

CCHS Depression Module

Currently, CIDI Short form for Major Depression (CIDI-SF) is used in CCHS

– Also used in NPHS and several regional and provincial surveys

Some problems with its use in CCHS– Has not been validated against International Classification of

Disease (ICD)– Evaluates 12-month prevalence, not necessarily current

treatment need– Does not evaluate some items related to clinical significance

Patient Health Questionnaire (PHQ) identified as potential CIDI-SF replacement

CCHS Depression Module

Primary goals of potential validation study:– Determine the validity of the CIDI-SF and PHQ in

relation to a gold standard diagnostic interview (SCAN – Schedules for Clinical Assessment in Neuropsychiatry)

– Identify optimal scoring procedures for the PHQ in Canadian population-based studies

CCHS Depression Module

Samples of n=200 subjects to be drawn in two sites (English and French)

– Supplemented with n=100 subjects selected from psychiatric outpatient settings to increase the number of positive cases of major depression

Each participant to be administered:– 1) Standard demographic module– 2) PHQ-9– 3) Module to distinguish between clinical depression and

bereavement– 4) SIDI-SF– 5) Set of modules to assess consequences of construct in

terms of quality of life

CCHS Depression Module

Sensitivity and specificity of the CIDI-SF and PHQ to be measured using the SCAN as a gold standard

Ordinal CIDI-SF ratings to be correlated with PHQ ordinal ratings using Spearman correlation coefficient

Test of construct validity of PHQ to be performed using exploratory factor analysis

Internal consistency of scales and subscales to be assessed using Cronbach’s alpha

Test-retest reliability of PHQ and CIDI-SF and inter-rater reliability of the SCAN will be evaluated for 50 respondents

CCHS Depression Module

The estimated cost for the project exceeded $200,000

Due to our inability to secure external funding and the lack of available budget and personnel internally, there are no concrete plans to proceed with study

Directions Forward

Focus on accuracy, interpretability and coherence Trade-offs between aspects of data quality

Improved timeliness, accessibility and relevance

How good is “good enough”? Partnerships

– Are there areas where CCHS, RRFSS and others can collaborate ?

Contact Information

Vincent DaleSurvey Manager, Canadian Community Health Survey

613-951-4265

Sylvain TremblayContent Manager, Canadian Community Health Survey

613-951-2528