Evaluation OVC AVSI

7/29/2019 Evaluation OVC AVSI

1/37

Evaluation as experience of knowledge

the OVC program in the Great Lake Region - Africa

itascabilipe r l a d i da t t i c a 11


2/37

All our knowledge, even the most simple, is always a minor miracle, since

it can never be fully explained by the material instruments that we apply to it.

Caritas in Veritate, n.77


3/37

Evaluation as Experience o KnowledgeAVSIs Program or Orphans and Vulnerable Children in the Great Lakes Region East Arica

2i

tascabili

11

This document contains the synthesis o the f nal evaluation o the AVSIs Program o Orphans and Vulnerable Children (OVC)

conducted by the Foundation or Subsidiarity (Fondazione per la Sussidiariet ), an Italian cultural and research no-prof t organization.

The OVC Program has been implemented by AVSI Foundation in Uganda, Rwanda and Kenya, rom 2005 to 2010, with the

f nancial support o the U.S. Agency or International Development and AVSI itsel.

This publication Evaluation as Experience o Knowledge: AVSIs Program or Orphans and Vulnerable Children in the Great

Lakes Region East Arica has been made possible thanks to a project called: Company or Development: Alianzas trans-

nacionales entre Actores No Estatles, Autoridades Locales y la comunidad institutional para una cooperacin al desarollo

ms ef ciente cod.DCI-NSA/2009/205-463 co-unded by the European Union.

The f rst objective o this project, started in November 2009, is to improve the inormation network and the exchange o best

practises o non governmental authorities and local authorities, involved in the AVSI network, also including the ollowing

European NGO: CESAL in Spain, VIDA in Portugal, AVSI POLASKA in Poland, FUNDATIA in Romania and SOTAS in Lithuania.

The opinions expressed in this document are o AVSI and do not necessarily refl ect those o the European Union.

Evaluation as experience o knowledge:

AVSIs Program or Orphans and Vulnerable Children in the Great Lakes Region East Arica

Pocket-edition n.11

Produced byBenedetta Fontana, AVSI

Authors: Lucia Castelli and Jackie Aldrette AVSI , with the collaboration o Carlo Lauro and Giancarlo Rovati,

Fondazione per la Sussidiariet

Copyright AVSI www.avsi.org year 2010

Cover Image Brett Morton

Graphic Design Accent on Design, Milan

Photocomposition and printing Pixart

ISBN Code 978-88-903534-9-9

AVSI USA

DC Of ce: 529 14th Street NW Sui te 994 Washington, DC 20045Ph/Fax: +1.202.429.9009 [email protected] www.avsi-usa.org

AVSI Italia

20158 Milano Via Legnone, 4tel. +39 02 6749881 [email protected]

47521 Cesena (FC) Via Padre Vicinio da Sarsina, 216

tel. +39 0547 360811 [email protected]


4/37

Evaluation as Experience o KnowledgeAVSIs Program or Orphans and Vulnerable Children in the Great Lakes Region - East Arica

3i

tascabili11

Index

1. Background and Methodology of the OVC Program 72. Achievements 13

3. Project Evaluation 17

4. Interesting related documents 21

5. Annex 1 23

6. Annex 2 33


5/37


4i

tascabili

11


6/37


5i

tascabili11

Backgroundand Methodology

of the OVC Program


7/37


6i

tascabili

11

6i

tascabili

11

Background and Methodology o the OVC ProgramChildren have been the main beneciaries o AVSIs activities since its arrival in Arica, 35

years ago. AVSIs concern is or the integral development o every child, with each oneseen as being endowed with inestimable dignity and potential, each unique and unre-peatable. For many years, AVSIs Distance Support Program (DSP) had made it possibleor thousands o children to attend school, have access to regular and emergency healthcare, and to know the care and concern o adults in their communities. Through the DSP,private donors are matched with a child who receives both direct and indirect support by

AVSI and selected partner organizations in his/her home community. A child is lookedupon as a member o a amily and community and AVSI always engages parents or guar-dians directly to enhance their responsibility towards their children. AVSI has traditio-

nally ocused on the most marginalized and vulnerable amilies and communities, withparticular attention to those aected by the HIV/AIDS epidemic.In April 2005, under the cooperative agreement with the U.S. Agency or InternationalDevelopment GPO-A-00-05-0002000, AVSI started implementation o a 5-year projectto expand its DSP intervention model across Uganda, Rwanda, and Kenya, later on to beextended to Ivory Coast.The overall goal o the project has been to improve the well-being and coping capacityo orphans and vulnerable children (OVC), as well as that o their amilies and commu-nities. AVSI chose to work both directly and via sub-grants to local community-basedorganizations, together with capacity building and mentoring.During the early stage o the program, AVSI decided to change the OVC acronym torefect more accurately AVSIs view o the programs purpose and vision; Orphans andVulnerable Children was thereore replaced by Our Valuable Children, thus emphasi-zing the positive resource inherent in every child and the community responsibility inrom o them and ownership over the program.Since the onset the project, AVSI embraced a amily-centered and community-basedmodel o care and support or OVC that relies on each individual child as an entry pointto a amily. This approach recognizes that every child despite his/her condition is unique,valuable and with special needs; it recognizes that the amily is very central or thegrowth, education and development o the child and that the community plays a vitalrole in nurturing the child and in supporting amilies.


8/37


7i

tascabili11

7i

tascabili

L a g o

V i t t o r i a

UGANDA

HOIMA

KITGUM

GULU

S U D A N

KEN

Y A

RU

AN

DA T

A

N

ZA

NI

A

T AN

ZA

NIA

R.

D

.

C

O

N

G

OPADER

KENYA

LIRA

MUTUATI

MULOT

KAMPALA

NA I ROB I

RWANDA

K IGAL I

HUMURE

Geographical Areas o intervention


9/37


8i

tascabili

11

8i

tascabili

11

In UgandaAVSI chose to implement the project largely through local partners, mainly in the

northern, central and western regions. The partners are local NGOs, FBOs, CBOs andschools within target communities.

In RwandaThe project covered 4 districts in the Eastern and Southern provinces. While in thebeginning AVSI mainly implemented directly with AVSI social workers, by the end o theproject six local partners were involved in direct activity implementation. The partnersare local NGOs, FBOs, CBOs and health centers within target communities.

In KenyaAVSI implemented the project in 10 districts spread across 5 provinces, namely Nyanza,Nairobi, Eastern, Central and Rit Valley provinces. Activities were carried out bothdirectly with AVSI social workers and through local partners. The partners are localNGOs, FBOs, CBOs and schools within target communities.

KAMPALA

NA IROB I

KIGAL I


10/37


9i

tascabili11

Achievements


11/37


10i

tascabili

11

The project reached a high number o benefciaries,

exceeding origanl target:

Uganda Kenya Rwanda TOT Description o benefciaries

41 20 6 67 partnersbuilt their capacity to oer quality and sustainable

services to OVC and their amilies

12,390 16,596 4,553 33,539 OVC who have been served with a wide range o services;

4,223 5,233 1,316 9,456households which have been strengthened

economically to provide or their members;

2,992 636 243 3,871caregivers (social workers, teachers, parents and

student leaders) who have been trained to enhance theirskills in OVC care and support

The basic or core needs o children and youth to which the project responded include:ood and nutrition, shelter and care, protection, health care, psychosocial support,education and economic strengthening or amilies. Results in specic core programareas are described in the table below.

Services/Indicator Uganda Kenya Rwanda TOT Comments/notes*

Food/nutritionalsupport (no. childrensupported)

3,974 5,089 692 9,755HIV positive children and those rom ood

insecure holds were served with ood

Shelter and Care (no.households supported)

6,188 10,379 4,485 21,052Children with no extended amilies andthose with households with difculties

in meeting shelter needs

Protection (no. childrensupported)

12,390 5,490 3,642 21,522Involves imparting o children rightsmessages to the community, birth

registration, inheritance rights issues

Health Care(no. childrenwho had accessto health care)

10,353 8,212 4,428 22,993

Agreement with hospitals, health carecenters and clinics, health insurance toprovide heath care to the supported

children and their amilies

Sensitization(no. contacts)

68,623 75,544 360,746 504,913Sensitization included topics on HIVprevention and behavior change

Psychosocial Support(no. children supported)

12,390 16,596 4,553 33,539

Psychosocial needs o children wereaddressed through ollow up support

visits by social workers and counseling tochildren and guardians

Counseling/communications (no.children supported)

12,390 5,693 3,292 21,375 Children and amilies with particularneeds received individual or groupcounseling by trained social workers


12/37


11i

tascabili11

Recreational activities(no. children supported)

10,843 10,647 43,290 64,780

Recreational activities include educationaltours, Christmas and birthday parties,

sport competitions, music, dance anddrama estivals, trips to recreational

centers

Education andVocational Training (no.children supported)

12,024 8,047 4,436 24,507 Children received schools ees, uniorms,

shoes, exercise and text books andtransport to acilitate their learning.

Quality education (no.

o schools supported) 66 71 53 190Rehabilitation o classrooms, provision odesks, Maps and atlases, educational kit

or nursery and primary schools,geometry ki ts or teachers, books orschool library, school eeding program

Quality education(no. o pupils in

supported schools)21,700 26,102 40,757 88,559

Quality education (no.o teachers in

supported schools)1,741 918 826 3,485

Care and support

(no. o caregiversteachers, socialworkers, trained)

2,992 636 243 3,871

Training and workshops organizedinclude: Observation and The HelpingProcess o OVC; Planning, Reporting

and Follow Up; Monitoring and

Evaluation; HIV/AIDS Prevention,Counseling and Testing; The Meaning

and Scope o Education; The NationalOVC Policy and Quality Standards,

Psychosocial Approach, Well being inclass, Introduction to Play Therapy

Economic Opportunity/Strengthening (no.o amilies supported)

4,223 5,233 1,316 10,772

Socioeconomic interventions includedtraining o caregivers in business skills ,

provision o start capitals and/or materia l(like seeds, animals), provision o start up

kits ater vocational courses (like toolboxes, sewing machines, saucepans etc)

Number o communityprojects

7 6 12 25Extension o piped water to the

communities that had no water or manyyears; renovations o houses or theneedy and elderly members in the

community; constructions o pit latrinesor amilies, road repair, houses

renovation, social halls renovations,toilets renovation, and ofce renovation

Number o amiliesbenefting romcommunity projects

9,000 7,580 5,860 22,440

Adult literacy(no. o adults)

760 778

Adult literacy courses were organized orcaregivers to improve their capacity toollow the children in their home worksand to be more equipped to start a

business


13/37


12i

tascabili

11


14/37


13i

tascabili11

Project Evaluation


15/37


14i

tascabili

11

Project EvaluationIn the nal year o the project, a longitudinal study o the beneciaries and partner

organizations was completed through the implementation o a nal household andorganizational survey. The survey data proved to be extremely rich in detail and inor-mation. The tools and processes were also useul to improve the capacity o AVSIand her partner organizations in each country to develop indicators, adapt evaluationinstruments and ultimately to collect standardized data on the projects impact.At the level of the child, most indicators related to education, psychosocial,health and nutrition all improved over the life of the project.Statistical analysis o the interactions among variables led to useul ndings. Oneimportant conclusion was the direct impact that guardians health and economic stabi-

lity has on the well-being o the child (considered holistically in terms o education,health and psycho-social indicators o well-being). In particular:children who exhibit bad general health are oten related to the guardians withbad health and to the outcome o poor perormance at school;children who report poor nutrition (less than 2 meals per day) are more likely tohave guardians who are illiterate or aected by bad health conditions;bad school perormance is more common among children with illiterate guar-dians, those who live in dirty houses or those in rural areas;on the positive side, children who maintain regular school attendance are morelikely to live in places that are less isolated, enjoy good health and nutrition, havea ather as guardian, live with guardians in good health and enjoy positive rela-tionships with the guardian.

The ndings conrm that indeed a amily centered approach is more eective inpromoting a childs well-being as opposed to simply child centered interventions.

A comprehensive summary o the evaluation done on children is in Annex 1.At the level of the partner organizations, the results of the study show overallimprovement in organizational structures and the awareness and skills relatedto quality service delivery to children and families. Among the results, theollowing conclusions can be noted, and linked to the work o capacity buildingdone by AVSI throughout the project1.

A general improvement o sel-awareness about the unctions o the board and thedierent managerial styles;an improvement o the relationships with the amilies or caregivers o the children;an increase in the degree to which they value networking and sharing o experienceswith other organizations; andmore attention given to sensitization activities.

A comprehensive summary o the evaluation done on partners is in Annex 2.Resulting from the analysis of indexes derived from both surveys (children and

organizational partners), we gain the most interesting and impressive results:

1 See http://www.avsi.org/documenti/AVSIConceptPaperCapacityBuildingOVC_care.pd


16/37


15i

tascabili11

There is a strong correlation between the Well-Being Status Index and the AVSIintervention with variables derived rom the partners survey like the Planning and

Sensitization Index and the Development Capabilities and Evaluation Index. Thehigher these characteristics are among partners, the higher the result on the mainindexes o children status and the capability o intervention o AVSI partners.Big partner organizations have better governance capabilities, as well as greaternancial sustainability and structure, but not necessarily good results in the well-being o the children or in the level o AVSI intervention. The size o the partnersdoes not play the main role in determining better results or the children. There is someindication that medium sized organizations (11-40 employees) managed to achievebetter results on child well-being indicators and AVSI programming indicators.

The type o organization has an important role in the level o impact on child-wellbeing.

The organizations with better scores on the Well-Being Status Index and with higherlevels in AVSI Intervention are the educational centers while NGOs are strong in gover-nance capabilities and nancial innovation and sustainability. The aith and charitableorganizations are stronger in the governance capabilities and in structural complexity.

These ndings conrm AVSIs approach in identiying local partners organizationsembedded in the communities with the main criteria o a common vision towards thegood o the child. The capacity building o local partners is very eective in promo-ting a childs well-being and will remain one o the core interventions o AVSI OVCprogramming.


17/37


16i

tascabili

11


18/37


17i

tascabili11

Interestingrelated documents


19/37


18i

tascabili

11

Interesting related documentsHere below a list o links to download the documents produced during the lie o the

project, including the ull report o the evaluation done on children and on partners.

http://www.avsi-usa.org/docs/pd/stories%20o%20hope.pdhttp://www.avsi-usa.org/docs/pd/FacesOHope.pdhttp://www.avsi.org/documenti/AVSIDidacticGamesForChildrensWellbeing.pdhttp://www.avsi.org/documenti/AVSIConceptPaperCapacityBuildingOVC_care.pdhttp://www.avsi.org/documenti/AVSIConceptPaperOnVulnerability.pdhttp://www.avsi.org/documenti/AVSIEligibilityCriteria-ScoringForm.pdhttp://www.avsi.org/documenti/AVSIVulnerabilityChildForm.PDF

http://www.avsi.org/documenti/AVSIVulnerabilityFollowUpForm.pdhttp://www.avsi.org/documenti/report_children_AVSI.pdhttp://www.avsi.org/documenti/OVC_partners_report_nal_AVSI.pd


20/37


19i

tascabili11

Annex 1


21/37


20i

tascabili

11

Executive Summary External Evaluation o ChildrensWellbeing, AVSIs Program o Support or Orphans and

Vulnerable Children in the Great Lakes Region- East Arica

October 2010In 2005, AVSI Foundation embarked on the endeavor o scaling-up its existing system osupport to the most vulnerable children and orphans in poor communities in Kenya, Ugandaand Rwanda, all o which have been directly aected by the HIV epidemic. This scaling-upwas made possible due to complementary nancing rom USAID through PEPFAR andwas easible because o AVSIs established network o local partner organizations.The main objective o the program has always been to increase the capacity o amilies and

communities to care or the most vulnerable orphans and children in HIV/AIDS aectedcommunities. In harmony with the expectations o USAID, AVSI sought to work towardsthis objective with a combined approach o direct provision o services to OVC and, indi-rectly, through a method and specic interventions designed to increase the capacity oamilies and communities to care or all vulnerable children. Thereore, AVSI utilized a seto objectives and corresponding indicators o outcome to guide the program along twoparallel tracks: directly improving the well-being o children and increasing the capacity oamilies and communities to care or children.In order to grasp the impact o its method on childrens wellbeing and on the organiza-tional capacity building o its partners in the three countries o the program, and in concor-dance with USAIDs strong interest in evidenced-based programming, AVSI contractedthe Foundation or Subsidiarity (Fondazione per la Sussidiariet), an Italian cultural andresearch no-prot organization, to conduct an evaluation exercise. The research teamcombined expertise in sociology, statistics, and eld based data collection. With input romthe project team, the researchers designed two surveys, one directed towards the domaino children and the other at the domain o partner organizations. Two sets o evaluationreports resulted rom this exercise.

Summary o MethodThroughout the AVSI OVC project (2005-2010), three surveys o a random sample oparticipating children were conducted by the external research rm together with AVSIsta based locally in Uganda, Rwanda and Kenya. The study utilized a longitudinalpanel survey design with the intention o capturing change among a stable, representa-tive, sample o participants over time. The three surveys were conducted in the springmonths o 2006, 2007 and 2009. Interviews were directed towards guardians and socialworkers o children (ages 6-16 years) in the sample o 1,200 children, taken rom theuniverse o children (2-18 years) enrolled in AVSIs OVC program at the start in 2005.2

2 The researchers decided to ocus on school-age children because o the program ocus on education andeducational outcomes. The age range was limited to those age 16 at the beginning o the program to allowor the ollow-up surveys to be conducted with the same children. The program eventually assisted over12,000 youth directly in the three countries.


22/37


21i

tascabili11

This Concluding Report documents the changes observed between data rom the 2006baseline survey and the 2009 survey with the longitudinal approach. In this way, a dynamic

prole o the changes occurred in each beneciary o the OVC program can be obtained.It is divided into three parts: (1) Outputs of Longitudinal Analysis, (2) Outputs ofMultivariate Analysis, and (3) Conclusions and Recommendations.The Longitudinal Analysis provided an in-depth prole o the children and amilies whowere selected or and participated in the project. The prole is dynamic, showing indi-cations o change over the period o project intervention and study. The resulting data,ocused on univariate statistical analysis disaggregated where relevant by country and sex,has been captured in ve dierent areas:

Child situation (orphanhood) and vulnerabilities

Guardian Characteristics and Vulnerabilities

Household CharacteristicsSchool Attendance and Perormance

AVSI Support Provided

For each o these areas, the data was synthesized into indexes which allow or quickcomparison o related data points according to the main programmatic objectives.3The indexes were:

Orphanhood IndexChild Nutritional IndexChild Health IndexGuardian Health IndexCare Giving IndexProperty IndexSocial Risk IndexSchool Perormance Index

The Multivariate Analysis involved the application o some advanced statistical tools toanalyze the relationships among multiple variables and build composite indicators. Thegoals were also to identiy segments o the population with similar characteristics, whichwould be applicable or the adaptation o program interventions or particular sub-target groups, and to understand the mechanisms o impact or prediction and planningpurposes. The tools utilized and reported upon in this report include the ollowing:

Factorial AnalysisCluster AnalysisChildren ProlingStructural Equation Modeling

3 More inormation and detail on the creation o these indexes and their component parts can be ound inCastelli L., Oliva F., Rovati G., Aldrette J., The Challenge o Evaluating a Project to Support Education orOrphans and Vulnerable Children: The Case o the AVSI OVC Program in the Great Lakes Region East

Arica, in Journal o Educat ion For International Development, 3:3, 2008: pp1-18.


23/37


22i

tascabili

11

The nal model approach allows or an estimation o the causal relationship betweenthe so-called maniest, or observed, variables collected through the survey and the

latent variables in order to identiy the drivers o change that have greater eect on thestatus and evolution o children. The analysis was guided by an interest in providingdata useul or the management and design o program interventions.

Summary o FindingsOverall, the research has provided an in-depth understanding o the children withinthis program.

By Longitudinal Analysis

Starting rom children vulnerabilityThe Orphanhood Index shows that 20% o the children have had a worsening o theirsituation due to the loss o a parent; more children are orphans (partial and total) thanin 2006. Among 6% o children, the index has improved, meaning that absent parentshave returned to the amily unit.

To improve Health Care and Health ConditionsThe Child Health Index shows that health conditions have remained bad or very bador have worsened or more than hal o the children. At the same time, health condi-tions are improving or 38% o the children (see Figures 6, 7). Results were slightlyworse in Rwanda and better in Kenya. Among the children, 61% have never beentested or HIV/AIDS, although this level is signicantly lower in Rwanda where only25% have never been tested. Overall, 3.8% o the tested children are HIV positive:5.6% in Uganda, 2.1% in Kenya, and 1.2% in Rwanda (see Table 5).Guardians health perormance also continued to be very bad during the period andworsening or the same percentage o guardians as those whose health is improving(see Figures 18-20). The rate o HIV/AIDS inection among guardians has increasedrom 14.9% in 2006 to 16.7% in 2009, while Uganda continues to suer rom thehighest inection rate o adults, at 24% in 2009 (see Table 12). Rates o HIV testingare signicantly higher among guardians than among children, around 65% comparedto 33%. Guardians poor health is increasingly aecting their capacity or work (seeTables 14, 15).

To improve Access to Food and NutritionThe Child Nutritional Index has clearly improved, with slightly better results orthe children in the urban areas. Rwanda continues to be the country with relativelyworse nutrition; 2009 data were collected during a period o drought conditions (see

Figures 2-5). The generally positive trends in the nutritional data lead to a preliminaryconclusion that AVSIs intervention helped to mitigate the impact o the ood crisis onthe participating OVC.


24/37


23i

tascabili11

23i

tascabili11

To improve Shelter and CareThe data refect slight improvements in shelter conditions and living density o the

OVC in the program. For example, the percentage o amilies living in single-roomhomes has reduced slightly rom 28% in 2006 to 24.5% in 2009. Indicators o housebuilding materials and access to water and sanitation continue to show a mixed situa-tion, with only slight improvements. The Social Risk Index captures indicators relatedto the living conditions and environment o the childs home; in 27% o the cases therisk actors have decreased and or 35% they have remained low (see Figure 29). Themost critical situation is ound in the urban slums where risks have increased morethan the average (see Figure 31).

To improve ProtectionChildrens involvement in work/labor is more widespread and requent than in 2006;90% o children work at some time, mostly in housework and digging, with rateshighest in Rwanda where 87% work every day. This output may be related to the actthat children have grown older, and conrms that according to tradition all amilymembers are supposed to contribute to household well-being. It has been importantor AVSI to note with concern that children belonging to households that receivedIGA support do work more than the average. Work, however, does not seem to havean impact on childrens personality or school perormance (see Figure 35); instead ithas a negative infuence on the relationship between the child and his/her guardian,possibly due to the amount o time spent together.

To improve Psychosocial Support and WellbeingIndicators o psychosocial wellbeing included child personality and relationships withguardians, adults and peers. A signicant percentage o children who were shy oraggressive in 2006 are now reported to be sociable and better adapted; this result islikely due in part to personal growth and AVSI intervention. The relationships o chil-dren with adults in their amilies have clearly improved, while the trend in terms orelationships with peers is less clear, with some 23% o children reporting worseningrelationships (see Tables 7,8 and Figures 12-14).

The Care Giving Index shows very good results in 2009 and clear improvement rom2006 with respect to the quality and quantity o time that children and guardiansspend together (see Figures 23-25).

To improve EducationConsidering school attendance and perormance o the children in the sample, veryencouraging results emerge. Although the children are delayed on their school path,

they have not accumulated any more delay during the time that they have spent inthe OVC program. Over 90% o the children attended school regularly throughout thelie o the project. The School Performance Index shows a generally positive trend;


25/37


24i

tascabili

11

children scoring badly disappear while 30% have improved and 36% have maintainedair or high perormance. Downward shits in perormance were mostly among those

children who transitioned rom primary to secondary school (see Figures 37-39).

Certain deciencies persist in terms o access to education and quality; children walklong distances and classrooms are oten overcrowded. A dierence is noted betweenthe average number o hours children spend in school and the availability o extra-curricular activities in Rwanda (where these indicators are lower) and Kenya andUganda (see Tables 53-55). The comparison o class ranking o the children consi-dered by country shows very positive results or the Rwandan school children despitethe quality indicators (see Table 65). At the same time, Rwandan students also demon-

strate the highest number o poor-perormers (see Table 66).

To improve Economic Strength o FamiliesMost amilies o OVC are involved in agriculture, casual labor, petty trade and services;Uganda has the highest number o adults engaged in more ormal capacities as civilservants, clerks or teachers. The sources o income or OVC amilies did not change.The Property Index shows improvement overall, though with notable distinctions bycountry. The most positive results come rom Rwanda. It should be noted that the surveyquestions related to household properties were weighted towards agricultural assets andlivestock, and thereore the relatively urban households in Kenya and Uganda may haveound less applicability in these questions.The Property Index produced better results or those households which received AVSIsupport or an IGA in 2006 (see Table 24). There has been an increase in household debtlevels over time, with households participating in IGAs also showing greater relevanceo household debt (see Table 27). Whether the debt-related indicators refect positiveor negative impact on overall household economic security, the data cannot provideconclusive evidence at this point.

By Advanced Statistical Techniques

Factorial and Cluster AnalysisThis analysis allowed or the identication o the actors aecting the survey results and

AVSI action as well as homogenous sub-groups o children within the sample basedon similar characteristics (Multiple Correspondence Analysis). These actors can mainlybe re-conducted to the living and economic condition o the amilies. Seven internallyhomogenous sub-groups emerged through the cluster analysis (see Part II, Figure 3)4.The cluster analysis conrms that children and their situation are really quite dierentbased on their place o origin and habitat (urban/rural). This conclusion suggests that

4 Table and fgure reerences in this section Advanced Statistical Techniques reer to those in Part II o the corre-sponding report, related to Outputs o Multivariate Analysis.


26/37


25i

tascabili11

25i

tascabili11

in order to improve outcomes, AVSI should dierentiate its interventions according toconditions specically related to each area and sub-group o children. More detailed

results by country and habitat can be ound in Section 8.3 o the report.

Children ProflingBy looking careully at the statistical prole o some critical categories o children, signi-cant associations among variables emerge. These relationships could be utilized whentargeting certain impact outcomes and designing uture interventions. Below are someexamples.

Children with HIV/AIDS are more oten related with the ollowing variables: maternalorphan, Uganda, habitat with high criminality, large amilies.

Children with bad general health

are more oten related with the ollowing variables:guardian bad health, large amilies, grandmother as guardian.Children who ate ewer than 2 meals a day are more oten related with the ollowingvariables: dirty housing areas, illiterate guardians, guardian bad health, and Rwanda.Children with bad school perormance are more oten related with the ollowing varia-bles: dirty house areas, houses without toilet, illiterate guardians, and rural areas.Children with unstable personalities are more oten related with the ollowing variables:bad hygiene, bad health, older guardians, areas o high criminality, and Rwanda.Children with aggressive personalities are more oten related with the ollowingvariables: paternal orphans, urban slums, guardian bad health, and Uganda.Children with shy personalities are more oten related with the ollowing variables: emales,under 13 years old, bad nutrition, and receive ewer visits rom social workers.

Further analysis and examples o this proling approach can be ound in Section8.4 o the report.

Structural Equation ModelingThis technique oers a measure o impact o AVSIs intervention on the childrenswell-being status and improvement, also taking into account the amily environmentand habitat. Two sets o outcomes were synthesized rom the data, a Wellbeing StatusIndex and a Wellbeing Improvement Index.

When compared to the previous years report, the Wellbeing Improvement Indexwas more infuenced by AVSIs direct interventions. However when looking at the

Wellbeing Status Index, it is almost equally dependent on AVSI interventions as onthe Family Environment Index. This highlights the importance o actors such as guar-dians health and home environment or the wellbeing o children.

The modeling exercise allowed or the creation o a Decision Support Matrix (seeFigure 8) which visually categorizes interventions into our quadrants: Area toMaintain, Area to Improve, Area to Monitor, and Area or Immediate Intervention. The


27/37


26i

tascabili

11

model highlights the importance or AVSI to maintain the level o services reached inrecent years given that results demonstrate high impact on both Wellbeing Status and

Improvement as measured by the respective path coecients in the structural equa-tion models estimated in this report. The set o Family Environment actors were theonly to all into the category o Area or Immediate Intervention, thus highlightingthe conclusion that household and amily conditions have a very signicant impacton the wellbeing o children (see Figures 9, 10). This conclusion is very important todrive a amily centered approach intervention.

Summary o Conclusions and Lessons LearnedProgram evaluations can be important opportunities or learning and capacity building

o implementers.The results represented and discussed in this report were made possible due to a numbero converging actors which acilitated the survey implementation. Among these were:

The commitment o the AVSI program and sta to the evaluation, including the pre-plan-ning necessary or a baseline survey.The programmatic context o having a large number o children in a homogenous situa-tion o need and vulnerability.The presence o a reliable on-the-ground network o people capable o data collectionthrough semi-structured interviews.

The research team took the decision to work closely with AVSI sta and collabo-rators both in the design o the instruments and the actual implementation o thesurveys. The research team believed that the advantages o this choice outwei-ghed the potential risk o positive prejudice towards the project and conditioningo interviews, as it ensured the involvement o highly motivated people with anexisting relationship with the children and amilies being interviewed, capable ounderstanding the responses received and thereore ensuring higher quality andreliability o data collected. The evaluation process turned into real action researchwhich resulted in a signicant, positive impact on the capacity o AVSI and partnerorganizations. Among the gains were: increased sel-awareness o tasks and respon-sibilities, increase appreciation or monitoring and data collection, more attentiveobservations o individual and collective needs and greater commitment to ndingeective solutions.

Program evaluations can and should produce data that is useul or management deci-sions to enhance programmatic impact.The survey process during the lie o the project allowed or program managementto make important changes in the ocus o the project. These included incre-

ased ocus on HIV testing and increased attention to guardian health and amilyresponsibility including economic capacity. Severe ood and nutrition needs wereidentied early and addressed.


28/37


27i

tascabili11

The longitudinal survey design allowed or conclusions that are generalizable across theprogram population as well as or customized evaluations o each single case.

The evaluation design served many purposes, one o which was to know the living condi-tions and changes o a sub-set o the population in much greater detail. The dynamiceatures o this design present an alternative to the use o a control group with each waveo data representing a term o comparison or the previous and subsequent wave.

The amily-centered approach is absolutely essential or having a direct and long-termimpact on childrens well-being and on specifc desired outcomes.Much o the data converged upon the conclusion that direct support given to a childis only part o the solution, and the amily environment and relationships must be

addressed in order to help the child to fourish. The results demonstrate the multiplelinkages among the ve strategic objectives o the OVC program: education, health, oodsecurity and nutrition, psychosocial support and economic security. For greatest impact,programs o this kind must be allowed to run or multiple years in order to see the ruitso the intervention and to allow or capacities to be built and relationships to blossom.


29/37


28i

tascabili

11


30/37


29i

tascabili11

Annex 2


31/37


30i

tascabili

11

Executive Summary External Evaluation o the OrganizationalPartners, AVSIs Program o Support or Orphans and Vulnerable

Children in the Great Lakes Region- East Arica October 2010In 2005, AVSI Foundation embarked on the endeavor o scaling-up its existing system osupport to the most vulnerable children and orphans in poor communities in Kenya andUganda, all o which have been directly aected by the HIV epidemic. This scaling-upwas made possible due to complementary nancing rom USAID through PEPFAR andwas easible because o AVSIs established network o local partner organizations.

The main objective o the program has always been to increase the capacity o amiliesand communities to care or the most vulnerable orphans and children in HIV/AIDS

aected communities.

In harmony with the expectations o USAID, AVSI sought to work towards this objec-tive with a combined approach o direct provision o services to OVC and, indirectly,through a method and specic interventions designed to increase the capacity oamilies and communities to care or all vulnerable children. Drawing on its yearso experience in the specic region and around the world, AVSI has understood thatcapacity building will never be achieved through a series o one-o interventions suchas training sessions or assessments, even with the best materials and trainers. Capacitybuilding happens through a combination o the practical aspects o working togetheron project management and administration, training on specic sets o skills, instil-ling the desire or improvement and growth, and education to the deepest levels omeaning or this particular work. Capacity building must begin rom a starting pointo respect and mutual trust, recognizing the local organizations or what they are:expressions o the society rom which they come and ormed by individuals withcertain values, knowledge, experiences, goals, and patterns o working.

In order to grasp the impact o AVSIs method towards capacity building, AVSIcontracted the Foundation or Subsidiarity (Fondazione per la Sussidiariet), an Italiancultural and research no-prot organization, to conduct an evaluation o the capacityo AVSIs organizational partners in Uganda and Kenya. The research team combinedexpertise in sociology, statistics, and eld based data collection. With input rom theproject team, the researchers designed a survey which was aimed at the partner orga-nizations with a space reserved or input by AVSI sta who had been working directlywith each partner organization. The project began in April 2005, and the PartnerSurveys were conducted in February 2006 and May 2009 among 48 partner organi-zations in Uganda and Kenya.5

5 Note that in 2006, partners in Rwanda were also surveyed. The conclusion rom the initial data analysis wasthat the context o AVSIs work in Rwanda and the characteristics o the local partners there diered so muchrom those eatures in Uganda and Kenya to make comparisons difcult and irrelevant.


32/37


31i

tascabili11

This report presents the data, analysis and preliminary conclusions rom the evalua-tion. It is divided into three parts: Outputs o Cross Sectional Analysis, Outputs o

Multivariate Analysis, and Conclusions and Recommendations.

Characteristics o Partner OrganizationsThis rst part presents an analysis o who the partner organizations are and the maineatures o their history and structure. What emerges is that the experience o AVSIin each country as well as the contours o civil society therein has had signicantinfuence on the shape o the organizations which became AVSIs partners on thisproject. For example, AVSI has had longer partnerships in Uganda and there is a widerrange o typologies o organizations, refective o AVSIs long history in the country and

the vibrant civil society. In Kenya, the partnerships are younger and more partners areaith based and specically religious congregations.

Over the lie o the project, the majority o partner organizations have demonstrated growthin terms o sta and volunteers. Considering the total outreach o the partners, an averageo 24% o their beneciaries are supported through this project (Uganda 31%, Kenya 13%).These ndings are one indication o increasing capacity and organizational stability.

The partners all provide a holistic package o services to children, consistent with AVSIsmethod and the project objectives; see Table 6. Education, health and psychosocial supportare the primary elements across the board. An interesting increase over the lie o the projectin the provision o economic strengthening activities to amilies and counseling and sensiti-zation to amilies has been noted in the data; see Table 8.

The sta o the partner organizations is relatively highly skilled; in Kenya 65.4% and inUganda 66.8% o employees have college or university level education. The segregation oroles and division o duties with the organizations depends on the size and type o organi-zation overall. The extent to which the board members and key sta share in the values andmissions o the organizations is relatively high; this commitment among volunteers is lessstrong, but has increased over the lie o the project. Specically the partner organizationsin Kenya demonstrated extremely high levels o personal commitment to the purpose othe organization; 90% expressed strong sharing o values in 2009, compared with 75% in2006; see Table 19.

Program Management by PartnersThe evaluation ocused on a ew proxy indicators to gauge commitment to the worko serving OVC and the eectiveness o AVSIs communication o methodology andquality service delivery. Among these indicators were the ollowing:

Type and prevalence o services delivered; consistency is noted with the AVSImethod and priority placed on education, health and psychosocial support ollowedby outreach to amilies in both countries.


33/37


32i

tascabili

11

Attention to the child through: Frequency and location o visits, see Table 21; Kenyan partners are able to reach

children more requently at their homes than in Uganda, where visits are requentbut occur most at school and at the ofce. Planning and decision making o visits, see Table 20; conclusions are mixed, but

planning o child visits seems to ollow urgency o situation and a balance o timingand geographic considerations.

Management o phase-out o children rom the project; data show that a muchhigher percentage (10%) o children in the project had been phased out over the

previous year in Kenya compared to Uganda (1.4%). Most oten partners selectedother as the reason or phase-out, presumably as a means o capturing a wide

variety o reasons based on each case.Vulnerability criteria guiding selection o children or project participation, seeFigure 6; consistency is noted with the AVSI method since the top criteria used be

partners were the ollowing, listed in order o importance: Single/total orphan Poverty HIV+ parent HIV+ parent

Partner Outreach to Parents and CommunitiesA specic section o the survey looked into the relationship o the partner organiza-tion with the parents and the community, including the community o local serviceproviders. The intention was to go beyond the capacities internal to each organizationto explore the external links which are essential or eective service o OVC in a long-term perspective.Overall, the data show a high level o trust o parents with the partner organization;80.6% in Uganda and 54.5% in Kenya. Yet, most partners expressed concern that ahigh level o parents may have stopped or reduced their commitment to care or theirchildren ater receiving support rom AVSI; see Table 23. Most partners estimated that20% o parents or guardians all into this category.

AVSIs partner organizations seem to have gained rom the experience o being linkedto one another within the ramework o this project. The partners expressed anincrease in the degree to which they value networking and sharing o experiences withother organizations. This opinion was also conrmed in the data showing that 56.5%o organizations have plans to continue these linkages over the next six months while22% are already doing so.The partners refect the capacity to sel-identiy organizational weaknesses and toormulate plans to address these weaknesses. The main developments either underway

or in planning are the ollowing, listed in order o importance:Search or other nancial resources and donorsManage transition o children leaving project


34/37


33i

tascabili11

Increase cooperation with other organizations and stakeholdersImprove relationship with local government

Increase sensitization work with amilies

AVSI Sta Evaluation o PartnersOverall, AVSIs sta gave a positive assessment to the capacity o partner organizationsand their improvement over the lie o the project in terms o administrative capacityand responsiveness.

A Multidimensional Data Analysis and ConclusionsIn this part, the report details the statistical methods which were utilized to analyze

the data rom the Partner Survey, summarized into ve area indicators (see Table 32),and compared with the results rom the Children Survey. In doing so, the resear-chers attempted to account or project results in terms o child well-being accordingto implementing organizations. The Children Survey oered data around child well-being, improvement o child well-being and strength o AVSI intervention, as capturedin the three indices, Well-Being Status Index, Well-Being Improvement Index, and

AVSI Intervention Composite Index respectively.

Secondly, a Principal Component Analysis resulted in actorial maps which allowedor a visual presentation o the data indicating direction and strength o relationshipsamong data points.

Thirdly, Cluster Analysis looked at the programmatic outcomes by partner organi-zations ater these were disaggregated according to the most essential characteristicsgrouped into clusters o similar organizations. This method divided the 48 partnerorganizations into our clusters as ollows:

Smaller partner organizations: these were ound to be less complex in terms ogovernance and structure but still producing results o child well-being consistentwith the mean values.

Main Ugandan partner organizations: these had outcomes higher than the meanor the Well-Being Status Index and the AVSI Intervention Index, and lower thanaverage outcomes on Well-Being Improvement.Big NGOs and FBOs: these larger organizations had very high governance andstructure results and high results on the AVSI Intervention Index, but loweroutcomes on the Well-Being Status and Improvement Indices.Big Kenyan Partners: these were relatively younger partners but older organiza-tions, with a high number o aith-based groups represented. Their results werevery high on structure, governance and nancial management, and outcomes were

very high on Well-Being Improvement, though less high on Well-Being Status.In summary, multiple indicators reveal important progress in terms o capacity o thesepartner organizations to serve their communities and in particular the most vulnerable


35/37


34i

tascabili

11

children and orphans. There does not emerge one type o organization or one sizewhich is best suited or most capable o ensuring improved well-being o children.

It does emerge that size o organization doesnt play the main role in determiningprogrammatic results. The type o organization is also not clearly or exclusively relatedto desired outcomes.These ndings conrm AVSIs approach in identiying local partners organizationsembedded in the communities with the main criteria o a common vision towards thegood o the child.


36/37


35i

tascabili11

From the same series

itascabili 1

Il bambino in situazioni di conitto.

Inglese/Italiano/Francese

itascabili 2

Educare il bambino, in amiglia, in comunit, nel mondo.

Inglese/Italiano

itascabili 3

The Challenge o HIV/AIDS: Twenty Years o Struggle.

Knowledge and Commitment or Action.

Inglese

itascabili 4

Educazione e lavoro nello sviluppo rurale. Esperienze da sei Paesi.

Inglese/Italiano/Spagnolo

itascabili 5

Unamicizia dellaltro mondo. Dieci anni di sostegno a distanza.

Italiano

itascabili 6

Arica Conitti dimenticati e costruttori di pace.

Italiano

itascabili 7

Argentina Valorizzazione della fliera della carne argentina.

Italiano/Spagnolo

itascabili 8

Capitale umano Risorsa per lo sviluppo

Inglese/Italiano/Spagnolo

itascabili 9

Haiti Germogli di speranza


itascabili 10Nutrire la persona, alimentare la speranza



37/37

AVSI Italia

20158 Milano Via Legnone, 4


47521 Cesena (FC) Via Padre Vicinio da Sarsina, 216


AVSI USA

DC Offi ce: 529 14th Street NW Suite 994 Washington, DC 20045

Ph/Fax: +1.202.429.9009 [email protected] www.avsi-usa.org

Documents

Evaluation OVC AVSI