Monitoring health system performance - synthesis of some

experiences from low-income countries

Dina Balabanova, Tim Powell-Jackson, Richard Coker, Kara Hanson & Anne Mills

London School of Hygiene and Tropical Medicine

Health System Metrics, Glion sur Montreux, 28-29 September 2006

Overview

Background Complexity Objectives and methods Measurement Health financing Health care delivery Emerging issues Conclusions

Background Commitment to invest in health systems is unprecedented,

but will not last unless it is possible to show results Currently poor health information available but demand for

improved health system metrics (national / international)

Opportunities – Health System Metrics and other initiative seeking to strengthen HIS– Commitment to the health MDGs – need to measure progress– Growing consensus of importance of measurement strategies &

monitoring & evaluation built into programme planning cycles

Threats – Limited resources for health information and sustainability– Capacity constraints (in the health and social sectors)

Objectives & Methods Purpose of the study: a review of some low-income

countries’ experiences with health system performance monitoring and use of data

Case study countries:– Georgia– Rwanda– Uganda– West Bengal, India– Material from other countries

Selection criteria Analytical approaches:

– uses the WHO health system performance framework– synthesis around common themes and issues– identifying unique lessons in each type of context

Complexity How should health system performance be measured?

– Increasingly multiple contacts with the system, chronic diseases

– Outcomes determined by different care components, sectors

– Need for system-wide and inter-sectoral indicators

Tension between international (donor-driven) demands and country-level agendas and needs

Use of normative approaches imply causality

To what extent monitoring influences policy?

Impact of measurement on health systems, e.g. Indicators that are measured often improve

Monitoring information may be complex to interpret where a range of interventions co-exist.

Measurement

What approaches are taken to measure health system performance in the study

countries?

What is measured ?Data Georgia Rwanda Uganda West Bengal

Demographic Census (2002) Census (2002) Census (2002) Census (2001)

Health financing NHA (2004) NHA (2003)

NHA (2001), public expenditure reviews,

Tracking Study (2001) NHA (2001)

Health outcomes incl. births and deaths

RHS (2005) & MICS (2006), Vital

registration & HMISDHS (2005), HMIS

(facility) DHS (2004) DHS (2005)

Co-coverage of interventions

MICS (2006) & RHS (2005) DHS (2005) DHS (2004) DHS (2005)

Human resources HMIS HR inventory HR inventory HMIS

Service provisionHMIS & SAM (2005-

06)

HMIS, SAM (2004) & Service Provision

Assessment Survey (2001)

HMIS, SAM (2004), Area Team assessments HMIS

Quality n/a n/a Various / accreditation n/a

Vertical programme monitoring

Immunisation, TB, HIV/AIDS

Immunisation, malaria, HIV/AIDS, TB etc.

Immunisation, malaria, HIV/AIDS, TB etc.

Malaria, RCH, TB, Leprosy, Polio, HIV/AIDS

etc.

Disease surveillance IDRSSentinel sites (HIV), early

warning system, IDRS Sentinel sites (HIV) HMIS

Health financing

How has information been used?Where are the gaps?

What challenges remain?

Use of health financing information

Identification of financing gaps and advocacy for increased allocation of funds to health (Rwanda)

Health sector leadership and management of funds (Tanzania, Rwanda)

Equity of health financing in the health system (South Africa, Rwanda)

Protection against the financial burden of ill health (Mexico)

Resource allocation with the health sector (Rwanda)

Gaps in health financing information

Private health expenditures – difficult to collect compared to public and external health financing sources

Coverage of NHA relatively low in developing countries but expanding

Health financing data at decentralised levels for local decision-making

Financial burden of ill health and impact on impoverishment at the household level

National Health Accounts in Africa

0 5 10 15 20 25 30

0

1

2

>2

NH

A R

ou

nd

s

Countries (N=46)

Number of NHA Rounds 1994 – 2004

Remaining challenges Institutionalisation of NHA into the routine activities of

Government

Underlying problems in Public Expenditure Management systems and data reliability

Timeliness of data (NHAs and household surveys)

Collection of private expenditure health financing data

Tension between disease expenditure and general health expenditure financial tracking

Addressing the needs of in-country policy makers vis-à-vis that of external agencies

Health care delivery

How has information been used?Where are the gaps?

What challenges remain?

Use of information: country examples

West Bengal, India

Aim: to monitor the performance of public sector programmes. Improve accountability and planning at national level

Standard service use indicators & regular meetings in PHC facilities

Uganda

Aim: to link health system performance monitoring to SWAPs and national policy process. Allows policy adjustment.

Data used in the annual health sector review process and to inform the development of annual plans

District league tables to rank performance of districts & motivate districts to improve indicators.

Tracking surveys – at the start of SWAP, 2001- to assess Govt systems (financial procedures, drug distribution, HR deployment)

Major gaps in measurement

Private sector – service use, service availability (infrastructure, human resources, services offered)

Vital events

Efficiency of health system

Quality of health care

Effective coverage

Remaining challenges

Low capacity and motivation to use data:

– Locally

– For decision-making or for policy initiatives

Lack of ownership by health providers, who are not involved in designing of monitoring procedure and indicators

Capacity for analysis concentrated at central level

Feedback to lower levels is limited, poor internal feedback

HMIS is often mistrusted

Selection of indicators often creates distortions

Information systems do not reflect move from project to system performance

– India: ‘critical milestones’ & vertical project indicators

Emerging issues

Data quality and reliability

Existing information systems, but data inaccessible or inappropriate to needs and policy process

Developing parallel monitoring frameworks rather than adapting & use of existing data: concerns for complexity and data reliability

HIS not always reflecting reform developments

Limited external data audit and reliance on single data sources (Rwanda, Uganda)

Technology involved in data collection, analysis and use often rely on bespoke software.

Parallel systems Donor agenda regarding data collection,

unsustainable

Data collection, analysis and use for policy is fragmented– Uganda/Nepal: lack of unified data linked to SWAPs– Private sector is often not covered (India/Uganda)

Multiple reporting requirements (Rwanda/India).

Lack of inter-sectoral information systems and unified quality standards. (Uganda/ Rwanda)

Vertical donors-supported programmes often function well in the short-term but may distort wider systems (e.g. Georgia & Angola)

Information flows & level of use

One-way traffic for information

– Disaggregated data not available at sub-national level

– Information intended to be used locally, is used at national level, or for different purpose reflecting governance & aid coordination

Information that is not aggregated nationally, less useful internationally

Governance and stewardship at local level needs to be able to draw effectively on aggregate & disaggregate data

– Disaggregated data feeds effectively into local planning when linked to decentralised decision-making (TEHIP)

Peer comparisons at district level – productive vs unhelpful

Factors facilitating measurement & use of data

Health system monitoring embedded within reform process– SWAPs/ PRSP in Uganda, Rwanda; district autonomy (TEHIP)

Unintended consequences (Afghanistan)– Selective use of data internationally (user fees/HIV, in Uganda)

In post-conflict settings, the aid influx promotes monitoring health systems & early warning systems. Possible inefficiencies.

The importance of governance– Channels for policy exist (annual reviews, SWAPs meetings) &

comparable timelines.

– Communities and non-health system stakeholders involved

Large-scale data collection exercises are resource-intensive and not synchronised with the policy process (some In-DEPTH/ LSMS).

Technology, appropriate to context

Conclusions

Effective health systems monitoring requires:

Capacity: to collect or use existing data, analyse, inform policy

Ownership

Coherence between domestic and external demands

Coherence between external agencies

Coherence between system-wide monitoring and vertical programmes performance measurement

Coherence between assessing the performance of different system elements

Domestic governance

Impact measurement to ensure sustainability/reform (scaling up)

Foster partnership between stakeholders

Acknowledgements

Georgia George GotsadzeIndia Barun KanjilalRwanda Vianney NizeyimanaTanzania Graham Reid Uganda Valeria Oliveira-Cruz

Freddy Ssengoba