Workshop on Energy Statistics, China

September 2012

Data Quality Assurance and Data Dissemination

1

2

Introduction

IRES Chapter 9: deals with Data Quality Assurance and Meta Data

Prerequisites of quality – institutional and organizational conditions, including:

Legal basis for compilation of data Adequate data-sharing and coordination between partners Assurance of confidentiality and security of data Adequacy of resources – human, financial, technical Efficient management of resources Quality awareness

3

Overview of Quality Assurance

Under IRES, countries are encouraged to:• Develop national quality assurance programs• Document these programs• Develop measures of data quality• Make these available to users

4

What is a Quality Assurance Framework?

All planned activities to ensure data produced are adequate for their intended use

Includes: standards, practices, measures Allows for:

• Comparisons with other countries• Self-assessment• Technical assistance• Reviews by international and other users

5

Quality Assurance Framework (Statistics Canada) Six Dimensions of Data Quality, based on ensuring “fitness

for use”

1. Relevance

2. Accuracy

3. Timeliness

4. Accessibility

5. Interpretability

6. Coherence

6

Quality Measures and Indicators

Should cover all elements of the Quality Assurance Framework

Methodology should be well-established, credible Must be easy to interpret and use Should be practical – reasonable, not an over-

burden For Key Indicators, see IRES Table 9.2

7

Promoting Data Quality at Statistics Canada Quality is a priority of senior management Key quality indicators are tracked Quality assurance reviews are conducted for major surveys Data quality secretariat established Questionnaire Design Resource Centre established Quality assurance training delivered Mandatory training provided to new employees

Quality assurance must be built into all stages of the survey process

Survey Stages:

1. Identification of needs

2. Survey design

3. Building the survey

4. Data collection

5. Data processing

6. Analysis

7. Dissemination

8. Archiving

9. evaluation

8

Quality Assurance Framework

1. Identification of Needs

Activities: Define objectives, uses,

users Identify concepts,

variables Identify data sources and

availability

Quality Assurance Consult with users and

key stakeholders Check sources for quality,

comparability Gather input and support

from respondents Establish quality targets

9

2. Survey Design

Activities: Design outputs Define variables Design data collection

methodology Determine frame &

sampling strategy Design production

processes

Quality Assurance Consult users on outputs Select & test frame Design & test questionnaire Test workflows Develop checklists Develop processes for error

detection

10

3. Building the Survey

Activities: Build collection

instrument Build processing system Design workflows Finalize production

systems

Quality Assurance Focus test questionnaire

with respondents Test systems for

functionality Test workflows Document

11

4. Data Collection

Activities: Select sample Set up collection Run collection Finalize collection

Quality Assurance Maintain frame Train collection staff Use technology with built

in edits Implement verification

procedures Monitor response rates,

error rates, follow-up rates, reasons for non-response

12

5. Data Processing

Activities: Integrate data from all

sources Classify and code data Review, validate and edit Impute for missing or

problematic data Derive variables Calculate weights

Quality Assurance Monitor edits Implement follow-ups Focus of most important

respondents Analyze and correct

outliers

13

6. Data Analysis

Activities: Transform data to outputs Validate data Scrutinize and explain

data Apply disclosure controls Finalize outputs

Quality Assurance Track all indicators Calculate quality indicators Compare data with

previous cycles Do coherence analysis

14

15

Sample Quality Indicators From IRES Table 9.2, linked to QA Framework Relevance: user feedback on satisfaction, utility of

products and data Accuracy: response rate, weighted response rate, number

and size of revisions Timeliness: time lag between reference period and

release of data Accessibility: number of hits, number of requests Coherence: validation of data from other sources

7. Data Dissemination

Activities: Load data into output

systems Release products Link to meta data Provide user support

Quality Assurance Format, review, test

outputs Produce and follow

dissemination checklists Ensure all meta data is

available Provide contact names

for user support

16

8. Archiving

Activities: Create rules and

procedures for archiving and disposal

Maintain catalogues, formats, systems

Quality Assurance Periodic testing of

processes and systems Ensure meta data is

attached

17

9. Evaluation

Activities: Conduct post mortem

reviews to assess performance, identify issues

Quality Assurance Consult with clients about

needs, concerns Monitor key quality

indicators Periodic data quality

reviews Ongoing coherence

analysis Investments

18

19

Meta Data

Important for assessing “fitness for use” and ensuring interpretability

Required at every step of the survey process Critical for enabling comparisons with other data Should include results of data quality reviews IRES table 9.3: generic set of meta data

requirements

20

Dissemination

IRES Chapter 10 – Dissemination Countries should have a Dissemination policy:

Scope of data available Reference period and timetable Data revision policy Dissemination of meta data and quality reports

Data collected should not be withheld Users must be aware of the availability Data must be accessible – barriers must be

reduced (e.g. format, cost, complexity)

21

Ensuring Confidentiality

Individual data must be kept confidential Complicating factors: small numbers of

respondents, dominance of a respondent Methods of protecting confidentiality:

Aggregation Suppression Other (e.g. rounding)

22

Balancing Confidentiality & Disclosure

An ongoing challenge and trade-off (relevance) Strategies to maximize utility:

• Raise the level of aggregation• Data which are publically available are fully used• Request permission to disseminate from respondents• Employ passive confidentiality• Publish confidentiality rules where data can be

disseminated provided “excessive damage” is not caused to the respondent

23

Reference Periods and Timetable Users must be aware of availability & release dates Reporting should be based on calendar year (Gregorian) Release targets recommended by IRES:

Monthly data within 2 months after reference period Quarterly data within 3 months after reference period Annual data within 15 months after end of reference period Key indicators should be released even faster

Ongoing challenge: the trade-off between timeliness, quality

24

Revisions Policy Countries should develop a revisions policy Provisional data should be revised when new or more

accurate data become available Two main types of revisions:

Routine revisions (e.g. for late reporters, corrections) Major revisions (e.g. changes in concepts, definitions,

classifications, data sources, sample restratification)

All meta data should be provided to support users in understanding the revisions

25

Dissemination Formats

Formats should be chosen to meet user needs Can be a combination of paper or electronic

formats Should always include meta data Should minimize barriers to access (e.g. cost,

technology, awareness, complexity)

26

Dissemination of Energy Data in Canada

Statistics Canada – primary data• All data are announced in the Statistics Canada Daily • Aggregate series available (free) on CANSIM• Major publications:

Report on Energy Supply and Demand in Canada (Energy Balances) Quarterly Energy Statistics Handbook

• Move from paper to electronic publications

Other major sources of energy information• Natural Resources Canada – energy efficiency indicators• Environment Canada – greenhouse gas emissions• National Energy Board – energy reserves, forecasts, trade

27

Thank you!

Andy Kohut, DirectorManufacturing and Energy DivisionStatistics CanadaSection B-8, 11th Floor, Jean Talon BuildingOttawa, Ontario Canada K1A 0T6

Telephone: 613-951-5858E-mail: andy.kohut@statcan.gc.cawww.statcan.gc.ca