Performance Audit

The GAVI Alliance Data Quality Audit experience

May 2008 ICGFM conference

Dr. Mercy AhunGAVI Alliance Secretariat, Geneva

Performance Audit• DQA history and rationale

• DQA overview

• Requirements

• Report Content

• GAVI DQA Experience

• Strengths of DQA

• Limitations of DQA

• Future Opportunities

History and rationale for DQA

• History: DQA emerged as an assessment tool to accompany the investment component of GAVI’s Immunization System Support (ISS) window

• Rationale: DQA aims to provide GAVI with assurance that the level of reward payable through ISS is based on a data validation process

Overview of Data Quality Audit

• Goal: assure that management of immunisation services are based on sound and accurate data

• Operational objectives:

1. assess the quality, accuracy, timeliness and completeness of immunisation reporting systems

2. audit the reported DTP3 coverage in infants in a specific calendar year and then estimate the National verification factor to use in reward allocations

Requirements• Countries receiving Immunisation Services

Support are required to conduct DQA during 2nd year of support

• External audit team engaged to review records & reports

• 4 districts randomly selected & data for previous year reviewed

• 6 Health facilities visited per district

• Number of vaccinations checked at facility, district & national levels

DQA report content• Performance indicators (DTP drop out rate,

wastage, coverage, Verification factor)

• Verification factor– Estimation based on all DTP3 records and reports

audited at facility & district level– Based on ratio of recounted DTP3 and reported

DTP3 (consistency )– Successful DQAs – VF of at least 0.80

• Quality of the system index: Composite index based on the recording, reporting, storage, M&E and system design for data

DQA reporting process

• In-country presentation

• Draft report with electronic copy

• Opportunities for feedback

• Confidentiality

• Monitoring of plan of action to address weaknesses in GAVI Annual Progress Report

GAVI DQA experience

Of the 52* GAVI-eligible countries that applied for ISS funding 2000-2007:

Number of Countries

Presented for review 43

Passed

Under review

* 9 countries did not require a DQA as ISS expenditure was below the minimum needed for review

39

4

Innovative: improving MIS

40

50

60

70

80

90

100

110

2002 2003 2004 2005 2006 2007

Average DQA 'verification

factor' (80=passing

score)

With introduction of Data Quality Audit (DQA), health data are becoming more reliable in all countries

Passed

All countries

Failed

78 11

5

415

1115

8

4

3

00 1

1

2

DQA strengths• Innovative Management, diagnostic and

capacity building tool – DQS – self assessment tool developed to

help countries diagnose and address data weaknesses

• Exposed need to improve skills in management of health data

• Induced improvements in repeat DQA countries• Enhances donor confidence and contributes to

transparency and accountability

Limitations of DQA

• Expensive & labour intensive

• Imprecision of verification factor– Wide confidence intervals

• Unable to capture verification of vaccination at point of delivery

Future opportunities for DQA

• Major evaluation underway to examine methods, sources scope

• Will review best practice and decision model

• Opportunities for expanding DQA to include other health system metrics

• Capacity building to enable local DQA or peer review process