View
216
Download
0
Category
Tags:
Preview:
Citation preview
2
Introduction
IRES Chapter 9: deals with Data Quality Assurance and Meta Data
Prerequisites of quality – institutional and organizational conditions, including:
Legal basis for compilation of data Adequate data-sharing and coordination between partners Assurance of confidentiality and security of data Adequacy of resources – human, financial, technical Efficient management of resources Quality awareness
3
Overview of Quality Assurance
Under IRES, countries are encouraged to:• Develop national quality assurance programs• Document these programs• Develop measures of data quality• Make these available to users
4
What is a Quality Assurance Framework?
All planned activities to ensure data produced are adequate for their intended use
Includes: standards, practices, measures Allows for:
• Comparisons with other countries• Self-assessment• Technical assistance• Reviews by international and other users
5
Quality Assurance Framework (Statistics Canada) Six Dimensions of Data Quality, based on ensuring “fitness
for use”
1. Relevance
2. Accuracy
3. Timeliness
4. Accessibility
5. Interpretability
6. Coherence
6
Quality Measures and Indicators
Should cover all elements of the Quality Assurance Framework
Methodology should be well-established, credible Must be easy to interpret and use Should be practical – reasonable, not an over-
burden For Key Indicators, see IRES Table 9.2
7
Promoting Data Quality at Statistics Canada Quality is a priority of senior management Key quality indicators are tracked Quality assurance reviews are conducted for major surveys Data quality secretariat established Questionnaire Design Resource Centre established Quality assurance training delivered Mandatory training provided to new employees
Quality assurance must be built into all stages of the survey process
Survey Stages:
1. Identification of needs
2. Survey design
3. Building the survey
4. Data collection
5. Data processing
6. Analysis
7. Dissemination
8. Archiving
9. evaluation
8
Quality Assurance Framework
1. Identification of Needs
Activities: Define objectives, uses,
users Identify concepts,
variables Identify data sources and
availability
Quality Assurance Consult with users and
key stakeholders Check sources for quality,
comparability Gather input and support
from respondents Establish quality targets
9
2. Survey Design
Activities: Design outputs Define variables Design data collection
methodology Determine frame &
sampling strategy Design production
processes
Quality Assurance Consult users on outputs Select & test frame Design & test questionnaire Test workflows Develop checklists Develop processes for error
detection
10
3. Building the Survey
Activities: Build collection
instrument Build processing system Design workflows Finalize production
systems
Quality Assurance Focus test questionnaire
with respondents Test systems for
functionality Test workflows Document
11
4. Data Collection
Activities: Select sample Set up collection Run collection Finalize collection
Quality Assurance Maintain frame Train collection staff Use technology with built
in edits Implement verification
procedures Monitor response rates,
error rates, follow-up rates, reasons for non-response
12
5. Data Processing
Activities: Integrate data from all
sources Classify and code data Review, validate and edit Impute for missing or
problematic data Derive variables Calculate weights
Quality Assurance Monitor edits Implement follow-ups Focus of most important
respondents Analyze and correct
outliers
13
6. Data Analysis
Activities: Transform data to outputs Validate data Scrutinize and explain
data Apply disclosure controls Finalize outputs
Quality Assurance Track all indicators Calculate quality indicators Compare data with
previous cycles Do coherence analysis
14
15
Sample Quality Indicators From IRES Table 9.2, linked to QA Framework Relevance: user feedback on satisfaction, utility of
products and data Accuracy: response rate, weighted response rate, number
and size of revisions Timeliness: time lag between reference period and
release of data Accessibility: number of hits, number of requests Coherence: validation of data from other sources
7. Data Dissemination
Activities: Load data into output
systems Release products Link to meta data Provide user support
Quality Assurance Format, review, test
outputs Produce and follow
dissemination checklists Ensure all meta data is
available Provide contact names
for user support
16
8. Archiving
Activities: Create rules and
procedures for archiving and disposal
Maintain catalogues, formats, systems
Quality Assurance Periodic testing of
processes and systems Ensure meta data is
attached
17
9. Evaluation
Activities: Conduct post mortem
reviews to assess performance, identify issues
Quality Assurance Consult with clients about
needs, concerns Monitor key quality
indicators Periodic data quality
reviews Ongoing coherence
analysis Investments
18
19
Meta Data
Important for assessing “fitness for use” and ensuring interpretability
Required at every step of the survey process Critical for enabling comparisons with other data Should include results of data quality reviews IRES table 9.3: generic set of meta data
requirements
20
Dissemination
IRES Chapter 10 – Dissemination Countries should have a Dissemination policy:
Scope of data available Reference period and timetable Data revision policy Dissemination of meta data and quality reports
Data collected should not be withheld Users must be aware of the availability Data must be accessible – barriers must be
reduced (e.g. format, cost, complexity)
21
Ensuring Confidentiality
Individual data must be kept confidential Complicating factors: small numbers of
respondents, dominance of a respondent Methods of protecting confidentiality:
Aggregation Suppression Other (e.g. rounding)
22
Balancing Confidentiality & Disclosure
An ongoing challenge and trade-off (relevance) Strategies to maximize utility:
• Raise the level of aggregation• Data which are publically available are fully used• Request permission to disseminate from respondents• Employ passive confidentiality• Publish confidentiality rules where data can be
disseminated provided “excessive damage” is not caused to the respondent
23
Reference Periods and Timetable Users must be aware of availability & release dates Reporting should be based on calendar year (Gregorian) Release targets recommended by IRES:
Monthly data within 2 months after reference period Quarterly data within 3 months after reference period Annual data within 15 months after end of reference period Key indicators should be released even faster
Ongoing challenge: the trade-off between timeliness, quality
24
Revisions Policy Countries should develop a revisions policy Provisional data should be revised when new or more
accurate data become available Two main types of revisions:
Routine revisions (e.g. for late reporters, corrections) Major revisions (e.g. changes in concepts, definitions,
classifications, data sources, sample restratification)
All meta data should be provided to support users in understanding the revisions
25
Dissemination Formats
Formats should be chosen to meet user needs Can be a combination of paper or electronic
formats Should always include meta data Should minimize barriers to access (e.g. cost,
technology, awareness, complexity)
26
Dissemination of Energy Data in Canada
Statistics Canada – primary data• All data are announced in the Statistics Canada Daily • Aggregate series available (free) on CANSIM• Major publications:
Report on Energy Supply and Demand in Canada (Energy Balances) Quarterly Energy Statistics Handbook
• Move from paper to electronic publications
Other major sources of energy information• Natural Resources Canada – energy efficiency indicators• Environment Canada – greenhouse gas emissions• National Energy Board – energy reserves, forecasts, trade
27
Thank you!
Andy Kohut, DirectorManufacturing and Energy DivisionStatistics CanadaSection B-8, 11th Floor, Jean Talon BuildingOttawa, Ontario Canada K1A 0T6
Telephone: 613-951-5858E-mail: andy.kohut@statcan.gc.cawww.statcan.gc.ca
Recommended