Quality Management of Statistical Processes Using Quality Gates Narrisa Gilbert Australian Bureau of...

Quality Management of Statistical Processes Using

Quality Gates

Narrisa Gilbert

Australian Bureau of Statistics

May 2011

Overview

• Background• What are Quality Gates• Attributes of Quality Gates• Six Components of Quality Gates • Outcomes of Quality Gates• Further information

Different perspectives on quality

ABS approach to quality management

• What?– Reinforcing a high degree of credibility for the ABS

and its outputs– Maintaining the relevance of ABS outputs

• How?– Effective relationships with respondents– Processes that produce high quality outputs– Regular review and evaluation of statistical activities– Staff skilled and motivated to assure the quality of

ABS outputs

Background• The Australian Bureau of Statistics produces a

lot of statistics

• On occasion the quality (accuracy) of data has been called into question at the last minute (right before release to the public)

• Sometimes no issues are identified• Sometimes errors are discovered

• Sometimes errors in data are identified after release

What are quality gates?

"checkpoints or decision points at various strategic places in a statistical

process at which the quality of the process at or up to that point is

explicitly assessed"

Attributes of Quality Gates

• Planned in advance of issues occurring

• Designed to facilitate the detection, discussion and resolution of issues and problems with processes earlier

• Constantly reviewed and revised

Attributes of Quality Gates

• Collaborative

• Communicate openly the progress/issues associated with the collections

• Provide a model of accountability and responsibility for survey processes

Attributes of Quality Gates

• Knowledge Management

• Create a store of corporate knowledge

• Document and monitor issues and actions throughout the collection cycle

Six components of quality gates

Placement Quality measures Roles Tolerance Actions Evaluation

1. Placement

Assigning priorities Related to risk profile of collection cycle Want to detect errors 'upstream‘

What can go wrong?

When can this problem occur?

What impact can it have?

Placement of Quality Gates

• For example

Within

Handover

UNECE (United Nations Economic Commission for Europe) Secretariat 2009, Generic Statistical Business Process Model, Version 4, April 2009, UNECE, Geneva.

Areas of risk changes to processes, systems,

methodologies

data transformations

overlap / coordination / integration with other areas

knowledge management

2. Quality measures Choose good indicators of potential

problems Prioritise! Definitions May need to drill down Time series useful

How would we know if this problem occurred?

3. Roles Owner area Sign-off person Operational person / Gate Keeper Stakeholders Gate definition Gate assessment / sign-off

Who is responsible?

Who will this affect?

4. Tolerance Driven by user requirements Consider size of natural variation, sampling

errors, level of detail, importance of outputs Need to form expectations Historical data may help Predetermined!

What is an acceptable level of quality?

5. Actions reflect extent of the problem reflect consequences "traffic lights" concept

What will we do (if there is a problem)?

Who needs to be informed?

6. Evaluation evaluation of processes evaluation of quality gates

What has this information told us about our quality?

How can we improve in the future?

Outcomes from Quality Gates

Business processes well-defined and known by staff

Responsibilities for processes and quality clearly defined

Statistical risk explicitly identified & assessed

Outcomes from Quality Gates

Explicit sign-off occurs at Quality Gates

Quality Measures clearly defined and understood

• Errors identified earlier (and fixed)

Further information

• Quality Management of Statistical Processes Using Quality Gates, Dec 2010, (cat.no. 1540.0) on the ABS website.

• http://www.abs.gov.au/ausstats/abs@.nsf/mf/1540.0

Questions

• Contact: • Narrisa Gilbert:

narrisa.gilbert@abs.gov.au