Principles, criteria and methods Part 1

Copyright 2010, The World Bank Group. All Rights Reserved.

Principles, criteria and methods

Part 1

Quality management

Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)


Definitions of quality/I

• Quality may be related either to subjective feelings or to objective facts

• Objectively, the quality of something is the sum of its essential attributes or properties.

• Quality has to do with user needs and satisfaction.• Users can not define the quality standards. • A minimum standard is to be transparent about the way in which

concepts and methods are used.

2


Definitions of quality/II

• Subjectively, something has quality when it meets the expectations of the user

• The International Organization for Standardization (ISO 8402) defines quality as: ‘Quality is the totality of features or characteristics of a product or service that bear on its ability to satisfy stated or implied needs of customers’.

• Openness about the data shortcomings and needed data revisions can help in getting understanding and support.

3


Quality frameworks

• Some international organizations have developed quality frameworks for official statistics

• These frameworks generally use the same kind of criteria, but there are slight differences

• One of the most comprehensive quality framework for statistics is the Data Quality Assessment Framework (DQAF), developed by the IMF

• Another useful international framework is the European Statistics Code of Practice (ESCP)

4


DQAF Quality dimensions

• The DQAF uses the following main dimensions of quality:

• Prerequisites of quality• Assurances of integrity• Methodological soundness• Accuracy and reliability• Serviceability• Accessibility

5


Prerequisites of quality

• Prerequisites are not quality criteria, but refer to institutional quality elements:

• Legal and institutional environment - the environment is supportive of statistics

• Resources - resources are commensurate with needs of statistical programs

• Relevance - statistics cover relevant information on the subject field

• Quality management - quality is a cornerstone of statistical work

6


Prerequisites – the indicators/1

• The responsibility for collecting, processing, and disseminating the statistics is clearly specified

• Data sharing and coordination among data producing agencies are adequate

• Individual reporters’ data are to be kept confidential and used for statistical purposes only

• Statistical reporting is ensured through legal mandate and/or measures to encourage response

• Staff, facilities, computing resources, and financing are commensurate with statistical programs

• Measures to ensure efficient use of resources are implemented

7


Prerequisites – the indicators/2

• The relevance and practical utility of existing statistics in meeting users’ needs are monitored and reported publicly

• Processes are in place to focus on quality

• Processes are in place to monitor the quality of the statistical program and reported publicly

• Processes are in place to deal with quality considerations in planning the statistical program and reported publicly

8


Assurances of integrity

• Explanation: the principle of objectivity in the collection, processing, and dissemination of statistics is firmly adhered to

• Elements are:

• Professionalism• Transparency• Ethical standards

9


Integrity – indicators/1

• Statistics are produced on an impartial basis

• Choices of sources and statistical techniques as well as decisions about dissemination are informed solely by statistical considerations

• The appropriate statistical entity is entitled to comment on erroneous interpretation and misuse of statistics

• The terms and conditions under which statistics are collected, processed, and disseminated are available to the public

10


Integrity – indicators/2

• Internal governmental access to statistics prior to their release is publicly identified

• Products of statistical agencies/units are clearly identified as such

• Advanced notice is given of major changes in methodology, source data, and statistical techniques

• Guidelines for staff behavior are in place and are well known to the staff

11


Methodological soundness

• Explanation: The methodological basis for the statistics follows internationally accepted standards, guidelines, or good practices

Elements:

• Concepts and definitions have to be in accord with internationally accepted statistical frameworks

• Scope must be in accord with internationally accepted standards, guidelines, or good practices

• Classification/sectorization are in accord with internationally accepted standards, guidelines, or good practices

• Basis for recording – Flows and stocks are valued and recorded according to internationally accepted standards, guidelines, or good practices

12


Methodology – Indicators/1

• The overall approach to statistics, in terms of concepts and definitions follows internationally accepted standards, guidelines, or good practices, as well as the scope, and the classification system

• The scope is broadly consistent with internationally accepted standards, guidelines, or good practices

• Classification systems used are broadly consistent with internationally accepted standards, guidelines, or good practices

13


Methodology – Indicators/2

• Market prices are used to value flows and stocks

• Recording is done on an accrual basis

• Grossing/netting procedures are broadly consistent with internationally accepted standards, guidelines, or good practices

14


Accuracy and reliability/1

• Explanation: Source data and statistical techniques are sound and statistical outputs sufficiently portray reality

Elements:

• Source data available provide an adequate basis to compile statistics• Assessment of source data• Statistical techniques employed conform to sound statistical procedures• Assessment and validation of intermediate data and statistical outputs -• Correct application of data collection, processing and data analysis.• Correct sampling• Correct application of the mathematical tools.

15


Accuracy and reliability/2

• Revision studies - Revisions, as a gauge of reliability, are tracked and mined for the information they may provide

• New methods may be needed to be introduced and existing methods may be needed to be revised and changed.

• Revisions also influence back-data.

16


Accuracy and reliability – Indicators/1

Indicators of accuracy and reliability include several aspects related to source date:

Data collection programs take into account country-specific conditions

Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required

Timeliness Routinely assessment Data compilation employs sound statistical techniques to deal with

data sources Other statistical procedures (e.g. data adjustments and

transformations) employ sound statistical techniques

17


Accuracy and reliability – Indicators / 2

Intermediate results are validated against other information where applicable

Statistical discrepancies in intermediate data are assessed and investigated

Statistical discrepancies and other potential indicators or problems in statistical outputs are investigated

Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes

18


Serviceability/1

• Explanation: Statistics, with adequate periodicity and timeliness, are consistent and follow a predictable revisions policy

Elements:

• Periodicity and timeliness - Periodicity and timeliness follow internationally accepted dissemination standards

19


Serviceability/2

• Consistency - Statistics are consistent within the dataset, over time, and with major datasets

• Revision policy and practice - Data revisions follow a regular and publicized procedure

20


Serviceability - Indicators

• Periodicity and timeliness follows dissemination standards

• Statistics are consistent within the dataset

• Statistics are consistent or reconcilable over a reasonable period of time

• Statistics are consistent or reconcilable with those obtained through other data sources

• Revisions follow a regular and transparent schedule

• Preliminary and/or revised data are clearly identified

• Studies and analyses of revisions are made public

21


Accessibility

• Explanation: Data and metadata are easily available and assistance to users is adequate

• Elements:

• Data accessibility - Statistics are presented in a clear and understandable manner, forms of dissemination are adequate, and statistics are made available on an impartial basis

• Metadata accessibility - Up-to-date and pertinent metadata are made available

• Assistance to users - Prompt and knowledgeable support service is available

22


Accessibility – Indicators/1

• Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts)

• Dissemination media and format are adequate

• Statistics are released on a pre-announced schedule

• Statistics are made available to all users at the same time

• Statistics not routinely disseminated are made available upon request

23


Accessibility – Indicators/2

• Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences from internationally accepted standards, guidelines, or good practices are annotated

• Levels of detail are adapted to the needs of the intended audience

• Contact points for each subject field are publicized

• Catalogs of publications, documents, and other services, including information on any changes, are widely available

24

Documents

Principles, criteria and methods Part 1