Impact Evaluation Conference Wageningen March 25, 2013 “Are the children better off”, a large scale concurrent evaluation in India Pam Baatsen, Senior

Impact Evaluation Conference Wageningen

March 25, 2013

“Are the children better off”, a large scale concurrent evaluation in India

Pam Baatsen, Senior Advisor SRHR and HIV, KIT

Amsterdam, The Netherlands www.kit.nl

Presentation overview

1. Purpose of the IE?

2. The intervention being evaluated?

3. Context?

4. Impact evaluated/findings?

5. Design IE and its validity?

6. Communication around findings?

7. Utilization of findings?

1. Purpose of the IE? Evaluation framework



2. The intervention? 2007-2012

11 Districts using a phased approach

40,000 households with over 73,000 children

Implementation by a consortium of 3 partners, coordination by government

Overall coordinationCommunity-

based HIV care and support

services

Government facility-based clinical services

Food Security/ Safety Net

KIT/Swasti

Evaluation


The Programme and Approach

Children and

families infected & affected

Psychosocial

Safety

net

NutritionEducation

HealthTo improve the quality of life of

children and families infected with and affected

by HIV

IMPACT: Decreased mortality of children living with HIV Decreased morbidity among children and parents with AIDS

Decreased number of children orphaned by AIDS Decreased number of children infected by HIV

FAMILY

CASE

MANAGEMENT

Amsterdam, The Netherlands www.kit.nlAmsterdam, The Netherlands www.kit.nl

3. The context • HIV highly stigmatized

• Regular unrest and insecurity

• Geographical and cultural differences

• Numerous changes in leadership within the State

• Absence of size estimates on # of children infected with or affected by HIV

• ½ state covered by this intervention and other 1/2 covered by another intervention

• Numerous other smaller interventions in all districts

• New antiretroviral therapy treatment guidelines over the course of the project

• Transitioning to government (after 5 yrs) donor requirement

Amsterdam, The Netherlands www.kit.nlAmsterdam, The Netherlands www.kit.nl

4. Impact evaluated?

Has the quality of life (QoL) of children infected and affected by HIV, and their households been improved?

Measured through: Changes in QoL scores; Body Mass Index (BMI), Mid Upper arm

Circumference (MUAC); trait hope scale; morbidity (outcome HIV tests, CD-4 count levels) and mortality (of children and or their parents)

And comparison between districts of this intervention versus other intervention


4. Impact evaluated and some findings…


• Mortality rate CLHIV higher than all children in the programme (4.6% versus 1%),

• However life of children living with HIV prolonged by the programme, including through successful earlier testing for HIV

4. Impact evaluated and some findings…

5. Design IE

Intensified Household tracking survey


Validity design? • Assumption 50% difference in the QoL of those reached versus

those not

• Generalization of survey findings to all the districts due to change in intervention protocol MUAC and BMI measurement not possible

• MIS important pillar for info (including mortality and morbidity) • The decision to collect QoL data on the basis of different

measurements, sometime contrasting findings but able to get overall picture

• Comparison state level data only possible for one (process) indicator



• Great effort to establish an evaluation core group at the onset of the evaluation to facilitate communication, and increase buy-in to the evaluation and its results

• Some guiding principles of evaluation protocol: • Credit to all those involved• Sharing of findings for action learning

• Sharing of the results through regular meetings; reflection action

learning workshops; and evaluation core group meetings

• Publication protocol: representatives of all parties to be involved . Reasons: 1) HIV data sensitive; 2) data could have a (negative) impact on the intervention and 3) contractual arrangements.



• Policy briefs prepared by evaluators – first round, however, these were not shared, as the INGOs were disappointed with the nil-finding.

• Also the end line findings, did not show the - by the implementing agencies - desired results: sharing of the final results sensitive

• Findings shared with everyone involved in intervention,

including at district level through a Reflection, learning and action workshop. However, it was left to the project implementers to share the findings with other key stakeholders in the state.


7. Utilization of findings?

• The IE has influenced the programme itself, to for instance better define the role of the outreach workers.

• Final results not shared with a large audience, due to the sensitivity of the findings.

• Efforts made to reach agreement on the publishing of an article with the main findings, but to date this has not yet been successful.

• KIT has advocated for the need to give more focus to younger children infected and affected by HIV on different occasions

Documents

Impact Evaluation Conference Wageningen March 25, 2013 “Are the children better off”, a large scale concurrent evaluation in India Pam Baatsen, Senior