PEPFAR

AIDS 2012 - Turning the Tide Together

PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011)Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author), Natasha Croome, Adele Daniels and Jaya Karira

Task: Summarize and Analyze PEPFAR OVC Evaluations

• 18 Evaluations• 22 Programs• 9 Countries – Kenya, Tanzania, Uganda, Mozambique, Rwanda, Namibia,

Zambia, South Africa and Haiti• 8 basic needs categories: Food/Nutrition, Health, Education, PSS, Economic

Strengthening, Child Protection, Shelter and HIV Prevention• Critique methodology and summarised findings.• http://resources.ghtechproject.net/content/pepfar-ovc-evaluation-how-good-doing-good

Evaluation – is it ever expedient?

1. Systematic reading by reviewers.2. Coding - extract themes agreed by

two reviewers. 3. Sub-analysis of costing.4. Methodology adequacy criteria.5. Seven basic needs categories. 6. Recommendations for future

evaluation and evidence- based policy.

Variable 1 2 Agree- ment

Refusal Rates 2 2 100%

Random Selection

2 3 66.7%

Gender 4 4 100%

Significance 4 4 100%

Comparison Group

3 3 100%

Cross Sectional 5 4 66.7%

Descriptive Data 7 7 100%

Double coding 6 reviews (Kappa =.83)

Evaluations Surveyed1. Brewster, Lee, et al, Catholic Relief Servies' OVC PEPFAR Programs: A Midterm Multicountry

Program Evaluation2. Nyangara, F., Obiero, W., Kalungwa, Z., Thurman, T., Community-based Psychological

Intervention for HIV-Affected Children and their Caregivers: Evaluation of the Salvation Army's Mama Mkubwa Program in Tanzania, USAID, AXIOS, Measure Evaluation, March 2009

3. Thurman, T., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., Nyangara, Community Education and Sensitization as an OVC Care and Support Strategy: Evaluation of the Integrated AIDS Program-Thika in Kenya, USAID, University of Nairobi Population Studies and Research Institute, Measure Evaluation, March 2009.

4. Thurman, T., Rice, J., Ikamari, L., Jarabi, B., Mutuku, A., Nyangara, F., An Evaluation: the Difference Interventions for Guardians Can Make: Evaluation of the Kilifi Orphans and Vulnerable Children Project in Kenya, USAID, Measure Evaluation, University of Nairobi Population Studies and Research Institute, March 2009.

5. Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T., Chapman, J., Promoting a Home-based Program Model for Supporting Children Affected by HIV/AIDS: Evaluation of Tumaini Project in Iringa Region, Tanzania, USAID, AXIOS, Measure Evaluation, March 2009

Evaluations Surveyed6. Swarts, Brian, et al, Empowering Better Care: Report on Economic Strengthening

for OVC Caregivers in Uganda, Summary of Findings, The Salvation Army World Service Office, WORTH, USAID and PACT, April 2010

7. Hutchinson, P., Thurman, T., Analyzing the Cost Effectiveness of Interventions to Benefit OVC: Evidence from Kenya and Tanzania, USAID, Measure Evaluation, April 2009

8. Nyangara, F., Obiero, W., Effects of a Community-Focused Approach Supporting the Mosts Vulnerable Children: Evaluation of SAWAKA Jali Watoto Program in Kagera, Tanzania, USAID, Measure Evaluation, August 2009

9. Thurman, T., Hoffman, A., Chatterji, M., Brown, L., A Case Study: Kilifi Orphans and Vulnerable Children Project, Measure Evaluation and Catholic Relief Services with USAID, March 2007

10. Nyangara, F., Thurman, T., Hutchinson, P., Obiero, W., Effects of Programs Supporting OVC: Key Findings, Emerging Issues and Future Directions

Evaluations Surveyed 11. Murombedzi, Cornelius, Mugumya, Firminus, Opportunity International/Habitat for Humanity:

Evaluation of Sustainable Income and Housing for Orphans and Vulnerable Children in Africa, International NGO Training and Research Centre, August 2009

12. Larson, B., Hamazakaza, P., Costella, J., Laurence, S., Costing of OVC Service Delivery in South Africa and Zambia, Center for Global Health and Development and Health and Development Africa, USAID/Project SEARCH, September 10, 2010

13. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Final Report, Christian Children's Fund and USAID, March 2008

14. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Z Scores Christian Children's Fund and USAID, March 2008

15. Scott, N., Pfaff, M., Semrau, K., Leyenaar, J., Suojanen, A., Mugala, T., Simon, J., End of Project Evaluation Report: Faith-based Regional Initiatives for Orphans and Vulnerable Children (FABRIC), USAID/Project Search, boston University OVC-CARE Project, Family Health International, August 2010

16. Yates, DeeDee, Finding the Inflection Point in Health Education and Economic Strengthening: Final Evaluation of Project HOPE's Track 1 OVC Program Sustianable Strengthening of Families and Orphans and Vulnerable Children Mozambique and Namibia, Cooperative Agreement #: GPO-A-00-05-00019-00, May 31, 2010

17. Capps, J., Bukuluki, P., Davis, S., Swarts, B., Bailey, B., Wakubona, S., Sustainable Community Support for Orphans and Vulnerable Children in Tanzania and Uganda, Final Salvation Army (TSA) Uganda, June 2010.

18. RAPIDS Evaluation Final Report 2005–2009 Key Findings; Population Council, Africare & Rural Net Associates, July 2009, Revised 2010

Overview and data abstraction

73

42

112

82

718

123

0 5 10 15 20

Refusal

Random selection

Consecutive selection

Baseline data

Power calc

Cross section

Ethical approval

Comparison groups

Theory

Number of studies

PEPFAR com

HIV OVC

Age IssuesFocus on Adolescents

• 8-14 (6)• 0-4; 0-18 (2)• 5-11 (1)• 7-12; 7-15 (2)• 13-18; 13-17 (2)• 12-19; 12-17 (2)• N/A – (2 costing studies)• 3 with unclear age

Grading methodology

• STRONG: No study used a RCT design in any form none graded as "strong".

• ADEQUATE: Nine studies graded "adequate" -some form of before/after, case control, and post intervention comparison design.

• 1 qualitative (with good design)• Two costing studies graded as adequate - utilised

consistent audit/costing data. • WEAK: Six studies were graded as "weak" and

conclusions/ findings should be viewed with caution.

Quantitative vs qualitative methodology

The war is over

• 13 used/included quantitative• 11 used/included qualitative • Qualitative is no excuse for poor methodology.• If qualitative used, quality standards on how sampled, how

recorded, how analysed – coding and themes – not just the use of quotes (which is what a number did)

Complex interventions• Most programmes are complex – mechanisms

of impact difficult to align.

Critical Absence of Baseline Data

Data Handling

• Vast number of data points (15 Scott)• Simplistic reporting (failure to do complex analysis)• Percentages versus statistical tests• Power calculations

Theory• Role played (low)• Recourse to theory (eg knowledge does not equate to behaviour – why focus and

measure of knowledge eg 9 Nyangara looks at knowledge gaining, some look at attendance at knowledge imparting (not even if they got the knowledge).

• Generalisability• Understanding mechanisms

Two forms of Evaluation – Against Pepfar Goals or against Programme goals

Methodological Trends

• Baseline: PEPFAR wants evidence-based approaches but most evaluations had no baseline, which is the only way to monitor and track impact of dollars and labor invested.

• Quantitative: Several of the evaluations were able to tell us how many OVC received services, how many caregivers had taken out a loan and repaid it.

• Standardized Indicators – good methodological reasons for using validated indicators (See Tulane group for good example).

• Outcome - How do you operationalize efficacy? (eg Yates 16 – use attendance at Health education as the indicator, but what if you attended and slept through?

• Gender: many mention few analyze

• Sampling – good methodological reasons for using excellent sampling and

reporting refusal rates.• Staff training in each year were a relatively large share of program costs. If such

expenses are considered investments in human capital, the training would provide services to the project over more than one year. If there is substantial staff turnover, prior investments in training would not provide services to the project in the future

• Dose Issue - Exposure dose seems to be an emerging issue. Home visiting is not a standardised commodity. Need to harmonise concepts in description and quantity standards

Findings by Basic Needs: Food/Nutrition - 14 / 18

• 9/14 evaluations focused on food/nutrition found OVC households to be moderately or severely food insecure.

• Improvements shown, but questions raised about dependence and sustainability.

Health Care ––11/18• 8/11 studies demonstrated improved health

outcomes• Specifically notable when combined with economic

strengthening and high dose/follow through

Education–– 15/18• Enrollment effect• Supplies positive mental health effect.• 0 / 14 identified gender imbalances in school

enrollment, although one reported that more girls than boys miss more than two days of school.

• Cost – resentment• Ongoing funding need until government shoulders

this.

Psychosocial––13/18

• 10/ 13 home- visits clearly benefited OVC but not always guardians

• Negatives associated with infrequent visits, low meeting attendance at Kids' Clubs

• Targeting carers indirectly benefits OVC• Volunteers overwhelmed by caseloads and ill equipped to

provide HB PSS counseling, particularly to adolescents and caregivers.

Economic Strengthening–9/18

• PEPFAR's investments in local savings and loan schemes are proving to be effective in raising many OVC households out of poverty

• Economic strengthening intersects significantly in other core areas of basic need, including food/nutrition, health, education, psychosocial support and shelter. Thus, economic strengthening remains one of PEPFAR's most potent and effective strategies for improving the lives of OVC and their families

• Costing - Great variation for example – Kids Clubs: $1.96 - $14.08 – Home Visiting: $3.55 - $89.68 – Gardening $24.00 - $89.00 – Food $1.39 - $32.36 – Educational Support: $17.23 - $141.32

Child Protection––16/18• Measurable role home-based visits play in

reducing child abuse, stigma and social isolation • Psychosocial interventions targeting guardians

often benefited OVC as well • Children whose homes had been visited by a

volunteer > birth certificate (36% of all children had birth certificates)

• Home visitor training associated with improved outcome

Shelter––6/18

• Shelter was the least prioritized basic need in all evaluations.• Provision of shelter made significant positive

differences in multiple areas, including protection, psychosocial support and health

HIV Prevention––15/18• Children's HIV knowledge appeared to increase in 7 out of 13

evaluations.• Some interventions appeared to be more thorough than

others.• School-based HIV prevention activities were part of 5 out of

13 prevention initiatives and were estimated to be cost effective.

Summary• Quality Standards. Define PEPFAR’s quality standards for evaluations?

Methodology? Standardization? The very mention of "PEPFAR" should be synonymous with quality

• Costing - Great variation• Volunteers - costly cyclical patterns are troublesome to program continuity and

quality, heavy reliance, questions of quality.• Food 3 / 13 focused on food - found > 80% of household samples were severely

food insecure (86%; 86.3%; 88.5%). • Kids’ Clubs – Mixed findings. Often volunteers poor results on issues of stigma and

discrimination.• One Stop Shop - Harmonizing and integrating complex PEPFAR-funded

OVC programs can yield better and multiple outputs • An emergency response should be saved for emergencies • Evidence- based programming should be a priority - There was weak

evidence of gains and impacts in the five core areas, but with a number of caveats

The way forward• Updated PEPFAR OVC Guidance (2012) stipulates: “Allocate

at least 10% of program budgets to M&E”• PEPFAR supporting the development and implementation of:

– Methodology to track impact of child welfare and protection systems strengthening

– Standardized quantitative program evaluation tool to measure child and caregiver/household outcomes across countries at population level (Will be introduced in Nigeria, Mozambique)

Evidence-Based Policy?

How Good at Doing Good?Reviewing the PEPFAR OVC Factor

andPEPFAR HKID Portfolio Review: Children of the HIV/AIDS Epidemic

Reports available at:http://resources.ghtechproject.net