LLIN Universal Coverage (UC) in Uganda:Process Evaluation (PE) Methodology

Presenter at KET: Caroline Asiimwe LLIN Process Evaluation Team (Adoke Yeka, Caroline Asiimwe, Thomas

Park, Wayne Stinson, Jenifer Peters)1

April 08, 2015

2

Outline• Background of the LLIN Universal Campaign (UC)• Purpose of Process Evaluation (PE) • Guiding PE Questions• Ideal Methodologies (International Standards)• Realistic methodologies (National Adjustments) • Pretest of PE Methodologies• Implementation plan for the PE• Qualitative and Quantitative Methods for the PE• Opportunities & Challenges of the PE design• Discussion

3

Background (Justification for the UC)

the WHO World Malaria Report on Uganda’s trends and effect of malaria morbidity & mortality

Endemic in 95% of UgandaPrimary contributor to poverty and low productivityMalaria control is a cornerstone of national developmentNational Development Plan 2010/11 – 2014/15, Health Sector Strategic Plan, 2010/11 – 2014/15LLINs play an important role in reducing malaria burdenMoH target - to protect at least 85% of the population at risk of malaria using measures that include LLINs.

4

Background (timelines for UC)

5

Purpose of the PE • The evaluation had two phases • Phase I - Process evaluation;–  To understand the extent to which the universal LLIN distribution campaign was implemented according to plan –  To evaluate the management of the program –  To review the design and instruments for the 2014 Malaria Indicator Survey (MIS) –  To document lessons for future comprehensive campaigns in Uganda and worldwide • Phase II – Impact Evaluation; • –  Determine the impact of the UC on malaria indicators • –  Provide accountability for the use of resources

6

Guiding PE Questions

• Relevance: Was the design and timing of the UC based on strong evidence?

• Effectiveness: To what extent did the UC attain its objective of national coverage

• Efficiency: Was the most efficient process adopted in implementing the UC?

• Impact: To what extent was LLIN ownership increased by the campaign?

• Sustainability: Is funding sufficient to carry out the UC every three years as planned

7

Ideal Methodology for PE (by RBM MERG)

Alignment to standards drawn by Roll Back Malaria M&E Reference Group (RBM MERG) and Alliance for Malaria Prevention

• Review of methods for post-campaign survey methodology (e.g Malaria Indicator Survey) – multi-stage randomised cluster survey recommended

• Review alignment to RBM MERG process indicators specific LLIN campaign strategy adopted

• Review justification for the selected indicator (i.e. howthe information will be used to improve campaign quality in future)

• Review data collection methods and timing (e.g supervisory visits, household visits and population-based surveys)

• Interview persons responsible for overseeing data collection and reporting • Review method for sharing the results (national post-campaign summary workshops,

coordinators meetings)

8

Ideal Methodology – PE Indicators by RBM MERG (1)

• Number and proportion of districts with functioning campaign coordination mechanisms in place

• Number and proportion of expected supervisory reports received at the district level

• Number and proportion of household registration form summaries completed correctly

• Number and proportion of planned radio spots broadcast

• Number and proportion of community mobilizers trained

9

Ideal Methodology - PE Indicators by RBM MERG (2)

• Number and proportion of households visited by a community mobilizer before the campaign

• Number and proportion of districts that received operational funds in time for the campaign

• Number and proportion of distribution sites properly equipped with trained staff

• Number and proportion of distribution sites with LLIN stocks correctly stored and accounted for

• Number and proportion of distribution sites that report a gap in stocks of LLINs

10

Realistic Methodology - Donors’ Approach to the PE

Evaluation system plan: Meta evaluation planned by the donors and MoH counterparts (Group critical evaluation sessions)

– Scope of Work Design: Engagement of MoH and non MoH implementing partners to scope the PE

– Sampling plan: (alignment to multi-stage cluster survey, with probability to proportionate size sampling) : 1st stage – national population, 2nd stage – district waves; 3rd stage – household individuals

– Measures for the PE: Efficiency, Effectiveness, Relevance, Sustainability

11

Realistic Methodology - MoH approach to the PE

• MoH expectations of the PE;a) detailed description of reality vs existing guidelines of the ideal b) measurement of two dimensions - the efficacy and efficiency of the operation c) recommendations to improve operational processes

• Intervention areas– Financing and financial management of the LLIN campaign– Procurement, transportation and storage of LLINs– Household registration and LLIN allocation– Distribution of LLINs to beneficiaries– Training and supervision– Advocacy, social mobilization and behaviour change communication

12

Pre-testing the PE Framework/Scope of work

• A pilot was conducted to determine a feasible PE plan; – Non participant direct observation of LLIN distribution in a purposive

sampled evaluation area– Content of UC– Beneficiary identification and registration – LLIN quantification– Rolling distribution versus mass coverage– Qualitative methods used to elicit (un)anticipated information, diverse

perspectives and Interests of different stakeholders

• Determine appropriate methods for quantitative and qualitative data collection (in context of cost, turnaround time, human resource)

13

PE Implementation Plan by Evaluators

• Mix of donor and MoH (National Coordinating Committee) preferred approaches– Qualitative Questionnaire Tool developed with MoH and

Donors, pre-tested in Mukono District – Review of MoH and Donor minutes and agreements– Used this information to map out target evaluation

informants and areas (purposive sampling)– Conveniently selected initial key informants, sub national

evaluation areas– Snowball (non probability) sampling applied to in-depth

interviewees (to elicit unanticipated information and represent diverse perspectives and Interests of different stakeholders)

14

PE for Planning & Coordination by MoH (1)• Appropriateness of UC Design

– Distribution method• Rolling Distribution/Waves MoH Preferred - 8

waves, each 15-16 districts, with a 24-day timeline per district)

• One time Mass– Categorization of UC;

• stand-alone (MoH Preferred)• integrated with other interventions

– Implementation strategy; • fixed site distribution (MoH Preferred)• house to house distribution

15

PE for Planning and Coordination by MoH (2)

– Beneficiary Identification and registration • household records (MoH Preferred), • vouchers/coupons; • wristband /bracelets

– LLINs Quantification • 1 LLIN/2 people, with rounded up/rounded

down odd numbers) MoH preferred;• No of sleeping spaces;• fixed number of LLINs/Household

16

Qualitative Methods used for the PE

– Interview tool used to guide interviews (Introduction, key, probing and closing questions)

– Telephone structured Interviews - planners at international level (The Global Fund)

– Face to Face structured interviews – planners at national level– Face to Face In-depth interviews – planners at sub-county and

community level – Focus Group Discussions – maximum diversity sampling

applied to community gatherings at LC meetings & health centres, with first hand experience with UC distribution points and LLIN use

– Data Coding based on pre-determined themes, analysis and reporting

17

Quantitative Methods used for PE (1)

• Effectiveness: National Coverage Evaluation considered;– Number and proportion of supervisors and community

mobilisers/distributors trained for UC– Number of LLINs procured and proportion supplied at

designated distribution points – Number and proportion of households registered and

received at least 1 LLIN per 2 people– Population targets (UBOS vs household registration)

Efficiency: Was the most efficient process adopted in implementing the UC? Were most activities implemented in a cost-efficient manner – PE did not carry out a detailed cost-benefit analysis, pended to Phase 2 Impact Evaluation

18

Opportunities realized with the PE methodology

• Provided a platform for actors to discuss and debate tools and actions (Meta-evaluation)

• Solicited expectations of the end user for the planners to review their plans for LLINs UC implementation- 2017

• Engaged other government departments to re-think public health issues and statistics (UBOS, State House, NGOs, Donors, etc)

• Enabled efforts to harmonize ideal and real applications of evaluation methodologies.

19

Challenges with implementing the PE methodology

• Unreliability of quantitative data (e.g. population discrepancies between UBOS and Household registered numbers)

• Lack of real time LLIN distribution data despite computerization of the UC system, to enable analyses

• Disaggregated databases (MoH, Implementing partners with different databases)

• Financial data hard to access or untimely• Parallel roadmaps between M&E and policy planners

(Evaluation methods not necessarily embedded in planning and politics)

20

End

Contact: karolasiimwe@gmail.com