SUMMATIVE EVALUATION OF IMPROVEMENT OF MOTHER AND CHILD HEALTH

SERVICES – PHASE II

INCEPTION REPORT

Not for quotation Curatio Imternational Foundation

10/04/2015

SUMMATIVE EVALUATION OF IMPROVEMENT OF MOTHER AND CHILD HEALTH

SERVICES – PHASE II

INCEPTION REPORT

October, 2015

This Report has been produced with the assistance of UNICEF. The contents of this report are the sole responsibility of independent experts and can in no way be taken to reflect the views of UNICEF.

MAP OF UZBEKISTAN

ACRONYMS

BCC Behavior Change Communication

CO Country Office

CP Country Program

DAC Development Assistance Committee

DR Desk Review

EMOC Emergency Obstetrics Care

EPC Effective Perinatal Care

ET Evaluation Team

EU European Union

FGD Focused Group Discussions

GDP Gross Domestic Product

GE Gender Equality

GGHE Global Government Health Expenditure

GIZ The Deutsche Gesellschaft fur Internationale Zusammenarbeit

GoU Government of Uzbekistan

HRBA Human Rights Based Approach

IDI In Depth Interviews

ILBD International Live Birth Definition

IMCHS Integrate Maternal and Child Health Services

IMR Infant Mortality Rate

JMT Joint Monitoring Team

KfW Kreditanstalt fur Wiederaufbau

MCH Maternal and Child Health

MCHCC Maternal and Child Health Coordination Council

MICS Multiple Indicator Cluster Survey

MMR Maternal Mortality Ratio

MoH Ministry of Health

NET Neonatal Equipment

NMCR Near Missed Cases Review

NR Newborn Resuscitation

OECD Organization for Economic Co-operation and Development

OOP Out of Pocket Expenditure

RoK Republic of Karakalpakstan

SP State Program

SV Site Visit

THE Total Health Expenditure

TOC Theory of Change

TOR Terms of Reference

TOT Training of Trainers

TWG Technical Working Groups

UN United Nations

UNFPA United Nations Population Fund

UNICEF United Children’s Fund

WB World Bank

WHO World Health Organization

1

TABLE OF CONTENTS

MAP OF UZBEKISTAN I

ACRONYMS II

1. INTRODUCTION 2

1.1 PURPOSE OF THE INCEPTION REPORT 2 1.2 INCEPTION REPORT STRUCTURE 2

2. COUNTRY CONTEXT 3

2.1 THE COUNTRY BACKGROUND 3 2.2 SOCIO-ECONOMIC AND DEMOPGRAPHIC CONTEXT 3 1.3 HEALTH AND HEALTH SYSTEM OVERVIEW 4 1.4 MATERNAL AND CHILD HEALTH 7 1.5 GENDER EQUALITY 11 1.6 PROJECT DESCRIPTION 12 1.7 KEY PROJECT STAKEHOLDERS 15

2. EVALUATION PURPOSE AND METHODOLOGY 16

2.1 EVALUATION RATIONALE, OBJECTIVES AND SCOPE 16 2.2 TARGET AUDIENCE FOR THE EVALUATION REPORT 16 2.3 GUIDING PRINCIPLES AND OVERALL APPROACH 17 2.4 EVALUATION CRITERIA AND FRAMEWORK 17 2.4 EVALUATION PROCESS 18 2.5: PROPOSED ASSESSMENT SCHEDULE AND EVALUATION TEAM MEMBER RESPONSIBILITIES 20 2.5.1 PROPOSED ASSESSMENT SCHEDULE 20 2.5.2 EVALUATION TEAM MEMBERS’ RESPONSIBILITIES 20 2.5.3 EXPECTED ROLES AND RESPONSIBILITIES FROM THE COMMISSIONING ORGANIZATION

AND OVERSIGHT COMMITTEE 21 2.6 METHODS OF DATA COLLECTION AND ANALYSIS 21 2.6.1 METHODS OF DATA COLLECTION 21 2.6.2 DATA SOURCES 23 2.6.3 METHODS OF DATA ANALYSIS 24 2.6.4 DATA QUALITY ASSURACE 24 2.6.5 EVALUATION LIMITATIONS 25 2.7 STAKEHOLDER PARTICIPATION AND ETHICAL ISSUES 26 2.7.1 PARTICIPATORY APPROACH 26 2.7.2 ASSURANCE OF INDEPENDENCE AND IMPARTIALITY 26 2.7.3 ETHICAL ISSUES 27

3: PRELIMINARY OUTLINE OF THE REPORT 28

ANNEXES 29

ANNEX 1: LIST OF DOCUMENTS REVIEWED 29 ANNEX 2: PRELIMINARY STAKEHOLDER LIST 31 ANNEX 3: EVALUATION FRAMEWORK 33 ANNEX 4: RESULTS FRAMEWORK 36 ANNEX 5: IDI GUIDE 41 ANNEX 6: FGD GUIDES 48 ANNEX 6.1: FGD GUIDES FOR SERVICE PROVIDERS 48 ANNEX 6.2 FGD GUIDE FOR BENEFICIARIES 50 ANNEX 7: TERMS OF REFERENCE 51

2

1. INTRODUCTION 1.1 PURPOSE OF THE INCEPTION REPORT

The inception report serves as the detailed agreed mandate and implementation plan for the Evaluation, providing for clearly agreed mutual expectations around the purpose and scope of the assignment. After the competition and award of the contract, an advance party of the selected Team carried out initial document research and preparatory work. This inception work has allowed the Evaluation Team to integrate the original terms of reference and its selected proposal into detailed plan of action to carry out the Evaluation. As part of the research and consultation in the inception phase of the Evaluation, the Team completed a focused “evaluability assessment” to verify in which areas of the IMCHS project Phase 2 of activities and its management fully or partially meet the preconditions for conducting a robust evaluation. This has informed how the Evaluation should be designed to be as reliable and useful as possible, given the available data, the context, the type of information the Evaluation is expected to convey, and the resources available. It has also helped to guide at which level (National, sub-national, local) each of the questions can best be assessed. The adjustments have been reflected in the detailed Evaluation work plan (including evaluation questions and matrix of evidence and methods) and the timetable.

1.2 INCEPTION REPORT STRUCTURE

This Inception Report has the following structure:

Chapter 2 describes the overall country socio-economic and health system context, brief situation analysis of the maternal and child health, equity related issues as well as provides brief description of the project and its key stakeholders. All these information is compiled based on the thorough desk review phase.

Chapter 2 explains purpose, objectives and the scope of the evaluation. It also describes evaluation criteria, framework, data collection and analysis methods and limitations, evaluation implementation phases, implementation plan and deliverables per each phase, as well as the role and responsibilities of each evaluation team member. The evaluation methodology has been largely informed by the evaluability analysis carried out during the inception phase and discussed and agreed with the UNICEF CO Evaluation Management Team.

Chapter 3 provides preliminary outline of the final evaluation report.

These three chapters are supported by Annexes, which include list of documents reviewed during the desk review, list of stakeholders to be interviewed, evaluation framework describing key questions, data collection sources and methods, Results framework, and interview and Focused Group guides for data collection. Finally annexes also include the original TOR of the assignment.

3

2. COUNTRY CONTEXT 2.1 THE COUNTRY BACKGROUND

Uzbekistan is a double landlocked country stretching 1500 km west to east and 1000 km north to south. It shares borders with Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan and Afghanistan. Formerly part of the Soviet Union it became independent in 1991. It has the largest population in Central Asia, officially 30.5 million as of 1 January 2014.The country is divided administratively into twelve regions, one autonomous republic (the Republic of Karakalpakstan) and the capital city (Tashkent). 2.2 SOCIO-ECONOMIC AND DEMOPGRAPHIC CONTEXT

Figure 1: Poverty Dynamics, 2001-2013

Uzbekistan is a lower middle-income country in Central Asia with up to 30 million populations, out of which 2.8 million are children under 5 years of age. According to national statistics, based on recent changes in calculation procedures by the State Committee on Statistics, the share of the population living in urban areas was 51.2% in 20121. Ethnic Uzbeks make up about 80 per cent of the population.

According to latest MDG report, the poverty rates2 decline, albeit regional variations continue to persists. An alternative multidimensional assessment of poverty, which takes into account access to services (education, healthcare, and housing), could provide a more comprehensive picture. According to such a multidimensional methodology to measure poverty, developed by the University of Oxford estimated multidimensional poverty index in Uzbekistan was 0.008 (range 0-1) with 2.3% percent of poor people. Inequality among poor was estimated 0.023 (range 0-0.3 with higher the value, the greater the inequality).3 The rapid growth of per capita income was accompanied by the reduction of inequalities. Between 2001-2014, the GDP per capita (current US$) increased significantly from USD 456.7 in 2001 to USD 2,037 in 2014 4 . The real wages of the population doubled and income differentiation reduced. The World Bank database does not provide estimates for GINI index5 for Uzbekistan, the only source is MDG report for 2015. According to this source the GINI index dropped from 0.39 to 0.29 between 2001 and 2013. Targeted regional development programs were designed and implemented in order to improve the quality of life of people living in rural areas and regions lagging behind the country average. These programs were aimed at accelerating socio-economic development through small business and private entrepreneurship development, reforms in agriculture, and investments into projects that would contribute to employment generation.

1Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014 2 Poverty is calculated based on definition of 2100 kcal food consumption 3 Based on 2006 MICS estimates. Available at http://www.ophi.org.uk/multidimensional-poverty-index/ 4 The World Bank Database, http://data.worldbank.org/indicator accessed on August 31, 2015 5 GINI index measures the extent to which the distribution of income or consumption expenditure among individuals or households within an economy deviates from a perfectly equal distribution.

27.5

14.1 13.7

Poverty rate (%) GDP growth, (%)

4

Another factor that contributed to poverty reduction was increased social expenditures to provide people with guarantees for basic social rights. Protection of public health, especially maternal and child health was identified as a priority of health care sector development. In 2013 total health spending was estimated to account 6.1% of GDP. Government Health spending from total Health spending ranged from 47.6% to 51.0% during 2001-2013. Public sector expenditure on health as % of total Government Expenditure is experiencing steady increase reaching 9.7% in 2013 however the spending is relatively lower compared to other CA countries such as Kazakhstan and Kyrgyz republic.6 1.3 HEALTH AND HEALTH SYSTEM OVERVIEW

Leadership and Governance: Under Uzbekistan’s Constitution every citizen has the right to skilled medical care7. The 1996 Law on “Health Protection” further protects this right8. Under the basic benefits package, introduced in the 1996 Law, all citizens have a right to primary and emergency healthcare, and care for “socially significant and hazardous” conditions. In addition population groups classified as vulnerable by the Government are entitled to a state guaranteed package of medical services free of charge. All medical services outside the package are financed by non-public sources.

The Uzbek health system has evolved from the Soviet Semashko model of health care and the public sector continues to constitute its core. The Cabinet of Ministers, which is accountable to the President and the Parliament, is at the top of the hierarchy of the health system both in terms of regulation and financing. It develops strategies, approves the health budget and holds other governmental agencies accountable for the implementation of health policies. At lower hierarchical levels, implementing agencies represents the Government. The Ministry of Health and the oblast (region) or rayon (district) health authorities assume administrative responsibilities, whereas the Ministry of Finance and its oblast branches (the oblast and rayon finance departments) are responsible for the implementation of financing directives. Although the administrative functions of the Ministry of Health and the oblast and rayon health authorities are tailored primarily towards the public sector, some of their functions extend to some degree to the private sector, such as the licensing of health care providers. The Ministry of Finance and its oblast and rayon branches, on the other hand, only deal with the disbursement and control of public funding to public providers of health care. Service Delivery9: a mixture of public and private health care providers forms the lowest layer in the hierarchy of the Uzbek health system. Public providers are tasked with the delivery of health care within a centrally set framework and can be divided into three categories, depending on their accountability and source of funding. Primary health care providers are administratively accountable to the rayon or urban health authorities and draw on public and private financing. Public financing to health facilities at the rayon level comes from the rayon or urban finance departments. The exception is the primary care units in the oblasts covered by the World Bank-financed “Health” project, which are financed from oblast finance departments. Private funding is obtained through the delivery of services outside the state-guaranteed basic benefits package of medical services. The next category of public health care providers is located at the oblast level. These are administratively accountable to the oblast health authorities and are financed through the oblast finance departments. These oblast health care providers include general or specialized hospitals and specialized outpatient clinics. The final category of public health care providers is

6 The World Bank Database, http://data.worldbank.org/indicator accessed on August 31, 2015 7 Constitution of the Republic of Uzbekistan, Article 40. The Constitution is available (in English), http://www.gov.uz/en/constitution 8 Available (in Russian) at http://www.med.uz/documentation/detail.php?ID=1028 9 Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014

5

located at the national (republican) level. A number of health facilities receive public funding directly from the Ministry of Health and are also administratively accountable directly to the Ministry of Health. Private providers, which are still small in numbers, are subject to the regulations of “for-profit” (profit-making) entities. Administratively, they are accountable to the local governments, while financial accountability lies with the local tax departments, to which private providers are required to submit regular financial reports. In addition to the statutory health system outlined above, some government agencies, such as the Ministry of Internal Affairs, national security services and major industrial companies, maintain their own health facilities. These parallel health care providers are directly accountable to, and receive funding from, the respective state agency or company. They primarily serve their respective employees, with little or no access by the general population. Public institutions exclusively provide medical education in Uzbekistan. They are administratively accountable directly to the Ministry of Health and the local and central governments, and are obliged to comply with the regulations of the Ministry of Higher and Specialized Education. Either the national or the local government provides public financing to these institutions10.

Although the national government conducts regular infrastructure evaluations and keeps an updated registry of inventories in public health institutions, investments are limited to minor repairs only due to the funding limitations11. Health facilities in general and especially rural primary health care facilities suffer from chronic underinvestment, as current spending on rehabilitation is significantly below the amortization rate12.

Financing: The allocation of resources for health care in Uzbekistan depends on the financing sources and the ownership of health care providers13. There are three principal mechanisms. In the first, public funding originates from the state budget (general taxation) and strictly follows the expenditure protocols developed by the central Government. Most of this funding flows into public health facilities, while a small share is directed towards the private sector, such as through the reimbursement for outpatient pharmaceuticals. In the second allocation mechanism, public health facilities draw on external funding. Public health facilities have been permitted to charge fees for services provided outside the state-guaranteed package of services. This funding might flow from a variety of sources, including out-of-pocket payments, employer contributions, or voluntary health insurance, and funding follows the protocols set by the central Government in a more flexible manner. In the third allocation mechanism, financing flows from external sources to the private sector, for which no protocols on expenditure and use of health resources exist. Table 1: Health Expenditure Dynamics (2000-2013)

Year Total expenditure

on health as a percentage of

GDP

GGHE as a percentage of

THE

GGHE as a percentage of

total government expenditure

Private expenditure

on health as a percentage of

THE

External resources for

health as a percentage of

THE

OOP expenditure

as a percentage of

THE

2000 5.3 47.5 8.7 52.5 6.7 52.3

2001 5.3 48.3 9.6 51.7 6.5 51.5

2002 5.4 44.6 6.5 55.4 1.8 55.2

2003 5.2 46.0 7.0 54.0 2.6 53.8

10 Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014 11 Ibid 10 12 World Bank, Republic of Uzbekistan Assessment of the Primary Health Care Reform: Transparency, Accountability and Efficiency, 20 May 2009, 13 Uzbekistan Health PPP – Improving Access to Medical Diagnostic Centers for the Poor, Project Information Document, The World Bank, 2011

6

2004 5.1 44.2 7.0 55.8 1.4 52.4

2005 5.1 44.6 7.3 55.4 1.8 52.1

2006 5.5 44.7 11.8 55.3 2.2 52.0

2007 5.8 39.5 11.1 60.5 1.4 56.9

2008 5.9 42.7 7.6 57.3 2.0 53.9

2009 6.3 41.5 7.6 58.5 1.5 55.0

2010 5.3 51.5 8.7 48.5 1.7 45.6

2011 5.6 50.7 9.0 49.3 2.0 46.4

2012 6.1 51.1 9.6 48.9 1.4 46.0

2013 6.1 51.0 9.7 49.0 1.7 46.1

Source: Global Health Observatory Database, accessed on 1.09.2015, http://apps.who.int/gho/data/

The public expenditure on health improved over the last 13 years measured by increased share of total health expenditure as a percentage of GDP (from 5.3% in 2000 to 6.1% in 2013) (Table

1). General Government allocates more public resources to health. If in 2000 only 47.5% of total expenditure on health was born by the general government, in 2013 this indicator increased to 51% in 2013, albeit the highest allocation was recorded in years 2006 and 2007 (before global economic downturn). In 2013 9.7% of general government budget was allocated to health sector representing 11% increase from year 2000. Private health expenditure as a percentage of the Total Health Expenditure (THE) remains high in Republic of Uzbekistan though shows declining trend (7%) since year 2000. Share of out-of pocket payments follows the same trend and accounts for 12% decrease in the period of 2000 - 2013. External funding to the health sector is shrinking and accounts to only 1.7% of the Total Health Expenditure in 2013 (Table 1). In a household survey conducted in 2007 by the World Bank, 42% of respondents reported to have made some kind of informal payment for health services. This included voluntary gifts (in kind or cash) or payments without a receipt. Informal payments impede the utilization of health services, in particular for the poor14. Expenditure on medication and medical supplies remain low, in 2011 making up only 3 per cent of the budget allocated to rural primary healthcare facilities, for example, with average spending per resident of 444 Sum (less than $0.50)15. This may be a factor behind shortages of essential medicines, equipment and hygiene materials, which are also highlighted as a concern by the UN Committee on the Rights of the Child. Health workforce: There is an urban rural divide in doctor coverage, with 36.3 doctors per 10,000 residents in urban areas, and 17-19.5 in rural areas16. As the host of national level facilities, Tashkent city has a rate of 76.8 doctors per 10,000 population, three times the national average. In addition, newly qualified medical professionals, tend to prefer to stay in urban and more central areas17. The Soviet legacy of greater respect for secondary and tertiary care, often make patients tends to seek medical services at higher levels18. Because transportation to these facilities tends to be more expensive, and in-patient treatment also costs more, this may lead to additional costs that affect the poorest and those from remote areas hardest.

14 Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014 15 WHO, Analytical Review of Primary Health Care in Uzbekistan: Achievements and Challenges, 2011 16 Ibid 15 17 Ibid 15 18 European Observatory on Health Systems and Policies, Uzbekistan: Health System Review, WHO, 2014

7

A World Bank survey report suggests that wages in the health sector cover only between 30 and 60 per cent of average living costs. As a result, some healthcare workers reported that they supplement their income with earnings from other sources19. Health information system: Each level of government provides reports on healthcare provision. The data are consolidated and analyzed by the appropriate Ministry departments, and then reported to Parliament, the Cabinet of Ministers, the President, and appropriate subunits and commissions20 . Access to health information is mostly limited to government agencies. Clear and well-structured mechanisms ensuring public access to reports and documents are not yet in place21. Evidence-based decision-making is still limited in both policy-making and clinical decisions. At the policy-making level, this is in part due to the unavailability of reliable and valid data to support the decisions policymakers are tasked with. However, there is also still little in the way of an evidence-based decision-making culture, or capacity to analyze and interpret available information. Lack of appropriate system-level data is also a major barrier to assessing health system performance in Uzbekistan22. 1.4 MATERNAL AND CHILD HEALTH

Healthcare reforms provided broad access for the population to quality, professional, well-equipped medical services, which include primary health care in rural areas, multi profile and separate specialized units in urban areas and regional centers, and high-tech specialized services in the capital city.

Maternal Health: The maternal mortality ratio demonstrates declining trend. Within the period of 2001-2013, the maternal mortality ratio (MMR) demonstrates decreasing trend, however mortality ratio by Interagency estimates and data obtained from the State Statistical Committee differ.

According to the Interagency estimates MMR declined from 48 per 100,000 live births in 2000 to 36 in 201323. Whereas according to official country statistics, MMR fell from 34.1 in 2001 to 20.0 in 2013, surpassing the national target of 22.6 per 100,000 live births, which had, been set for 2015 (Figure 2).

Figure 2: Maternal Mortality Ratio (2000-2013) and Mortality Structure (%) (2013)

19 World Bank, Republic of Uzbekistan Assessment of Primary Health Care Reform: Transparency, Accountability and Efficiency, 2009, https://openknowledge.worldbank.org/bitstream/handle/10986/3065/445300ESW0UZ0P1LIC0Disclosed0718191.pdf?sequence=1 20 European Observatory on Health Systems and Policies, Uzbekistan: Health System Review, WHO, 2014 21 Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014 22 Health Systems in Transition, Uzbekistan Health System Review, WHO, 2014 23 Global Health Observatory Data Repository, http://apps.who.int/gho/data , accessed on August 31, 2015

Source: Global Health Observatory Data Repository, accessed on August 31, 2015 and State Statistics Committee

4844

403634.1

29.2

21 20

0

10

20

30

40

50

60

2000 2005 2010 2013

per

100,0

0 l

ive b

irth

s

Maternal Mortality Ratio, 2000-2013

Interagency estimate State Statistics

22.2

20.6

10.36.3

9.5

10.3

18.3

Mortality Structure (%), 2013

Obstetric haemorrhageLate toxicosis

Infectious diseases CVD

Obstetric embolismSepsis

Other causes

8

Half of maternal deaths are directly related to obstetric factors, while indirect causes include pre-existing health conditions aggravated by pregnancy, such as anemia, heart disease, etc. According to 2010 WHO data24, around 49 per cent of maternal deaths are directly related to obstetric factors or incorrect management of complications (Figure 2). This is despite the high coverage of antenatal care and of skilled attendants at birth (recorded as 99 and 100 per cent respectively in MICS 2006)25. This suggests that the main causes may be related to the poor quality of health services provided. The indirect causes of maternal mortality may include pre-existent health conditions that are aggravated by pregnancy, such as anemia, heart disease, diabetes and HIV.

Regional maternal mortality also decreased, although remain higher than average in some regions. MMR decreased significantly in Jizzak, Kashkadarya, and Namangan regions. As of 2013, the MMR was above the national average (20) in Navoi, Bukhara, Khorezm, Fergana, Tashkent regions and the City of Tashkent26. It is important to note, that careful approach is needed while discussing the regional MMR data because the population data for this indicator is not enough to use the statistical approach to get accurate figures within the regions, where number of births does not reach the proposed number as per formula. Access to reproductive health services is universal. The coverage of pregnant women by antenatal care during the first 12 weeks of pregnancy increased from 87.9% in 2002 to 89.5 in 2013. Between 2000 and 2012, coverage of antenatal care increased substantially in Fergana, Samarkand, Namangan and Sirdarya regions. In 2013 the highest level of coverage had been achieved in Sirdarya, Navoi, Fergana, Jizzak regions. Use of modern methods of contraception was reported by 65% of women married women or in union according to the latest available Multi Cluster Indicator Survey (MICS) 2006 data (Figure

3). Compared to the MICS 2000 survey, there is a slight decrease at the overall level of use and modern methods. The most popular method is the intrauterine device, which is used by half of all married women in Uzbekistan. All of the remaining contraceptive methods have percentage not exceeding 3% implying clearly that IUD is the most widely preferred method in Uzbekistan.

The use of contraception is highest in the Easter region at 71% and the lowest in the Southern region at 56%. As a result of high levels of education among women in Uzbekistan, less differentiation is observed among different categories of education (Figure 3). Differentiation is less evident with regard to wealth status of the household and mother tongue of the household head.

Figure 3: Maternal Mortality Ratio (2000-2013) and Mortality Structure (%) (2013)

24 WHO regional estimates for CIS (1997-2007), 2010 25 Multiple Indicator Cluster Survey, 2006, UNICEF 26 MDG Report 2015, Government of Uzbekistan, UNDP, 2015

46

55.1

55.5

47.7

52.2

60.9

61.8

58.8

59.1

56

59.9

56.3

57.8

54.3

POOREST

SECOND

THIRD

FOURTH

REACHEST

UZBEK

RUSSIAN

KARAKALPAK

Contraceptive Prevalence rate

MICS 2006 DHS 1996

62.50%

64.40%

58%

59.30%

60.40%

56.80%

TOTAL

RURAL

URBAN

Contraceptive prevalence- modern methods (%)

MICS 2006 MICS 2000

9

Child Health: Introduction of the system of delivering and raising healthy children ensured considerable success in the reduction of infant mortality in Republic of Uzbekistan. As official figures show reduction in under five mortality rates from 28.5 to 13.8 per 1,000 live births between 2000 and 201227. Infant mortality (under one) fell from 20.2 to 10.2 per 1,000 live births between 2000 and 201228. According to the 2013 UN Interagency Child Mortality Report, under 5 mortality was estimated at 40 in 2012 as compared to 61 in 2000. The report also gives infant mortality at 34 and neonatal mortality at 14 per 1,000 live births (Figure 4). There are several potential reasons why figures differ. Firstly, Uzbekistan’s official statistics by year 2015 are calculated based on the Soviet Live Birth Definition (LBD), while the UN estimates use the International Live Birth Definition. Research has shown that use of the ILBD results in an infant mortality rate that is 20-25 per cent high eras compared to the same indicator calculated based on the Soviet LBD29. Notably, ILBD has been introduced as a formal reporting system countrywide only starting 1st July 2014. Secondly there appears to be in complete reporting of infant deaths. A 2007 WHO report suggests that the discrepancy between official data and estimates is due to the misreporting of births and infant deaths, partly due to the fear of negative consequences by medical personnel 30 . There is no recent published evidence available that this practice has changed. Figure 4: Dynamics of U5 Child Mortality Rate and Infant Mortality Rates (per 1000 live births) (2000-2013)

According to Interagency estimates, Infant Mortality Rate (IMR) almost halved from 46.7 per 100 live births in 2000 to 19.8 in 2013. Neonatal mortality (NMR) also declined by 58% since 2000 from 12.7 to 7.4 in 2013 as shown in the (Figure 4), albeit with lower pace compared to IMR and U5MR.

Infant mortality rates (IMR) are uneven and show disparities in the country. Official statistics of year 2012 indicate that infant mortality is 13 per 1000 in urban areas and 7.9 in rural areas, with

27 Data on Under 5 mortality for 2012 is from State Committee on Statistics, Socialnoe Razvitie I Uroven Zhizni v Uzbekistane, 2013 and for 2000 from State Committee on Statistics, Healthcare in Uzbekistan, 2007 28 Data on infant mortality for 2012 is from State Committee on Statistics, Zdravokhranenie Uzbekistana, 2013 and for 2000 Ministry of Health Institute of Health, Statistical Material on Paediatric Institutions, 2012.Citing State Committee on Statistics data 29 UNICEF, Right At Birth: Birth Registration in Central and Eastern Europe & the Commonwealth of Independent States, 2008, 30 European Observatory on Health Systems and Policies, Uzbekistan: Health System Review, Vol.9, No. 3 2007, http://www.euro.who.int/__data/assets/pdf_file/0004/96421/E90673.pdf

Source: WHO, Interagency Estimates and State Committee on Statistics

28.5

20.6

14.8 14.1 13.8

61

55

50 49

40

0

10

20

30

40

50

60

70

2000 2005 2010 2011 2012

per

1000 l

ive b

irth

s

U5 Mortality Rates

National, official dataInteragency group

Pe

r 1

00

0 liv

e b

irth

s

Infant Mortality Rates

NMR IMR U5MR

10

Tashkent City having the highest IMR in the country31. The most recent survey, MICS 2006, reported higher infant mortality rates in rural areas (50 per 1000 live births) than in urban areas (44per 1000). For Tashkent city, MICS reported IMR at 34 per 1000 live births, 1.4 times lower than the national average (48 per 1000). In addition, the same source indicates, that child and infant mortality for the poorest quintile are 1.6 times higher than for the richest quintile32. Infant mortality is primarily caused by issues in perinatal care 59.3%, congenital malformation (18.1 %) and by respiratory diseases (3.4%). The main causes of under five mortality are respiratory diseases (26.4%), issues in perinatal care (24.7%) and injuries and accidents (13.5%) and congenital malformations (9.7%)33. Key issues affecting infant health include the quality of care for the mother during pregnancy and her nutritional status both during pregnancy and in early life. A key concern is the burden of domestic labour placed on young brides in Uzbek society. As revealed in UNFPA 2012 study on child marriage, young brides assume responsibility for housework and serves, in essence, the husband’s whole family34. Another factor affecting child survival is the uneven distribution of skilled healthcare providers: as seen above, those living in Tashkent city and surrounding areas seem to have better access to highly qualified staff and better healthcare facilities than those in regions such as Kashkadarya, Surkhandarya, Karakalpakstan and Khorezm35.

Figure 5: U5 Child Mortality Causes (2013)

Source: Global Health Observatory Data Repository, accessed on August 31, 2015 and State Statistics Committee

Parenting practices, and especially the capacity of caregivers to recognize danger signs of childhood disease, are another possible reason for infant and child deaths. There is little recent data available but, for example, in the 2006 MICS (the latest available data), only 15 per cent of caregivers were able to recognize two symptoms of pneumonia, while this indictor was 3 per cent in the Ferghana Valley and 6 per cent in Surkhandarya and Kashkadraya Oblasts. In general, the percentage increased with increasing education level and socio-economic status36. According to official statistics in 2013 for most routine vaccines almost universal coverage has been reported. According to MICS 2006 only 81 per cent of children under five received all recommended vaccines. Coverage fell between first and subsequent doses for most routine vaccines37. More recent survey data on this is not available.

31 State Committee on Statistics, Zdravokhranenie Uzbekistana, 2013, Statistical Bulletin 32 Multiple Indicator Cluster Survey, 2006, UNICEF 33 Global Health Observatory Data Repository, accessed on August 31, 2015 and State Statistics Committee 34 Giorgio Tamburlini, CQI in MNC care in Uzbekistan: Report of reassessment of 4 Regional Maternity Centers, UNICEF, 12 May

2011 35 Figures from State Committee on Statistics, Healthcare in Uzbekistan statistical bulletin, 2012 36 Multiple Indicator Cluster Survey, 2006, UNICEF 37 Multiple Indicator Cluster Survey, 2006, UNICEF

26.4

24.713.5

9.7

25.7

1-59 months

Acute lowerrespiratoryinfectionsIssues inperinatal care

Injuries

Congenitalanomalies

All othercauses

3.4

59.3

3.9

18.1

15.3

0-27 days

Acute lowerrespiratory infections

Issues in perinatalcare

Injuries

Congenitalanomalies

All other causes

11

Figure 6: U5 Child Malnutrition 1996-2006

U5 Child nutrition status improves. There is no information is available whether large scale collection of child growth and development data, based on WHO recommendations, is taking place on a regular basis. Therefore, for nationwide anthropometric data the report refers to MICS 2006 and subsequent recalculations by UNICEF Headquarters. Comparative analysis of MICS data on underweight prevalence shows some progress between 1996 and 2006. Disparities among different groups of children seem to be considerable.

Children in the poorest quintile are twice as likely to be underweight and 40% more likely to be stunted as those in the richest quintile38.

GAIN’s 2012 household survey reported 34.4% prevalence of anemia among women of reproductive age nationally. Using WHO criteria, this level of prevalence indicates that anemia is a moderate public health problem in Uzbekistan as a whole. However, Surkhandarya Oblast and the Republic of Karakalpakstan had the highest prevalence of anemia, estimated at 45.6% and 44.2% respectively, indicating that anemia is a severe public health problem in these two regions. In 2006, 26% of children in the country were exclusively breastfed for the first six months (MICS 2006). According to data from the Scientific Research Centre of Pediatrics, the exclusive breastfeeding rate in the country is more than 90% on discharge from maternity facilities. However, the rate declines to 30%-50% in subsequent months39.

According to official statistics, coverage with Vitamin A supplementation reached 98% of all children under five in the country during the last five years40. If accurate, this builds on the already high coverage found by MICS 2006, which indicated that 90 per cent of children aged 6-59 months had received Vitamin A supplements. 1.5 GENDER EQUALITY

Appropriate legal and institutional frameworks created to promote gender equality in Uzbekistan. Uzbekistan has developed and is constantly improving the appropriate national mechanisms for the advancement of women at the level of the legislative and executive authorities. The regulatory framework of the Government's policy on gender equality was established in the national legislation (the Constitution, a number of decrees and resolutions), as well as by Uzbekistan's accession to a number of international initiatives (for example, the signing of the Beijing Platform for Action, accession to the UN Convention “on the Elimination of all forms of discrimination against women”, and the draft law “on equal rights and equal opportunities” that is pending approval). The appropriate national institution for mainstreaming gender and protecting women’s rights was created in 1991 with the establishment of the Women’ Committee of Uzbekistan, the Head of which was also appointed as Deputy Prime Minister by decree. The Chairpersons of the regional Women’s Committees were similarly appointed as the local Deputy Khokims (governors). A National Plan of Action to improve the status of women in Uzbekistan was

38 Multiple Indicator Cluster Survey, 2006, UNICEF 39 UNICEF, Infant and Young Child Feeding Programme Review Case Study: Uzbekistan, 2009 40 Ministry of Health data provided to UNICEF

18.8

7.9

4

31.2

21.119

0

5

10

15

20

25

30

35

UDHS 1996 UHES 2002 MICS 2006

Underweight Stunting

12

created at the end of the 1990s to provide direction to the Government, parliament, public organizations and civil society in their efforts to promote gender equality. A 2004 President Decree of the Republic of Uzbekistan further provided impetus to support the Women’s Committee in developing and implementing policies in the field of social and legal support to women, welfare, protection of health and opportunities to participate in public life. Opportunities for equal access to education, health care and employment supported through social protection policy. In order to facilitate women’s access to opportunities, a number of social assistance policies were implemented. These include: Benefits to non-working mothers for the care of a child up to 2 years old. The inclusion in labor contracts of benefits such as monthly financial assistance to women on maternity leave for babies from 2 to 3 years old and the reduction of working hours by one hour for women with babies under three years. The proportion of women in public office increased significantly, some of it thanks to the introduction of the quota system 41 . The proportion of women in the Executive Power increased 5 times from 3.4% in 2005 to 16% in 2013. The proportion of women in government also increased. As of November 1, 2013 the proportion of women in high level of position (Ministers, First Deputy Ministers, Deputy Ministers, Heads of the territorial government bodies, and etc.) Was about 11% and mid level positions (Heads of main departments, their deputies, heads of departments, etc.) was 31.2%. At the local level, women’s participation increased in institutions of local government and in the Makhalla structures. In 2013, the share of women who were elected as chairpersons of Makhalla increased from 9.6% in 2006 to 25.6% in 201342. Women’s share in employment increased. For the period of 2000-2013 the proportion of women’s employment increased from 44% to 45.7%. Women’s employment in small businesses and private entrepreneurship rose significantly. Women ran 40.4% of small enterprises and 13.7% of micro-enterprises (2012). As far as regional disparities are concerned, the best regions in terms of gender parity are Karakalpakstan, Andijan, Samarkand and Tashkent regions43. 1.6 PROJECT DESCRIPTION

In the past decade the country made a steady progress in improving health, nutrition and wellbeing of mothers judged by decline of Maternal Mortality and child mortality rates, albeit Uzbekistan will not be able to achieve MDG (4 and 5) targets. The progress is also evident in reforming overall health sector and maternal and child health system in particular, though challenges with lack of modern health management practices at all levels of the system and the quality of maternal and child health system care remain to be Government’s main concern. The process of health care reforms in Uzbekistan largely benefited from the external technical and financial assistance received from multi and bilateral donors. Among all development partners, the volume and size of EU assistance in MCH sector has to be acknowledged. Since 2008 till present EU funding was made available in support of MCH sector reforms in two phases. To address MCH service bottlenecks Ministry of Health (MOH), UNICEF and European Union initiated the “Improvement of Mother and Child Health Service - Phase I” (IMCHS I) program funded by the EU from 2008 to 2011.

If the first phase of the support to the “Improvement of Maternal and Child Health Services” (IMCHS) project mostly concentrated at expanding the application of the WHO Live Birth Definition, together with strengthening newborn care and improving the quality of maternal and child health care by developing skills and capacity in selected regions of the country, the second

41 Uzbekistan MDG Report, UNDP, 2015 42 Uzbekistan MDG Report, UNDP, 2015, MDG 3 43 Ibid 40

13

phase of the IMCHS project, aimed at supporting the Ministry of health to capitalize on the achievements of the Phase I, enhancement and expansion of the coverage of the quality maternal- infant health services and filling the systematic gap related to the management of the MCH health sector and low level of population knowledge on responsive parenting. Particularly the second phase of the project focused on enhancement of decision-making capacities of both, health care providers - in management of service provision that potentially contributes to the improvement of the service quality as well as families/households. At the family/household level informed decisions is thought to lead to better health outcomes after delivery and improved nutrition practices for (expecting) mothers and children.

Table 2: Key Project Information

Project title Improvement of Mother and Child Health Services in Uzbekistan – Phase II

Implementation period 25 July 2012 – 31 December 2015

Project duration 41 months

Total eligible cost € 5,718,770.00

EU contribution € 4,900,000.00

UNICEF contribution € 818,770.00

Overall objective

Contribute to human development in Uzbekistan with a special emphasis on Mother and Child Health Care (MDGs 4 and 5).

Specific objective

Support the Ministry of Health to increase the quality of mother and child health care services, and to increase the capacity of families to make informed choices about health and nutrition.

Expected results

Result 1: Institutional strengthening. The skills and operational capacity of the Ministry of Health and its three tiers (primary, secondary, and tertiary) have been enhanced to effectively support the health reform process concerning Mother and Child health services, in accordance with international standards. Result 2: Capacity development and empowerment. A comprehensive education-training process dealing with mother and child health care has been developed and put in place, at both a medical-institutional and a family level.

Target Oblasts

Fergana, Tashkent, Bukhara and Khorezm Oblasts, the Republic of Karakalpakstan, and Tashkent City

Target groups At national level: The Ministry of Health and Republican Medical Institutes At regional level:

- Four Oblast Health Departments (Fergana, Bukhara, Khorezm and Tashkent), the Ministry of Health of the Republic of Karakalpakstan and the Health Department of Tashkent City;

- 13,269 health care professionals (including 771 Trainers, 560 Managers, and 11,938 Physicians and Nurses);

- 12 Rural communities in pilot Oblasts

Final beneficiaries 15.5 million population of Fergana, Bukhara, Khorezm and Tashkent Oblasts, Republic of Karakalpakstan and Tashkent city, including 3.5 million children under the age of 5 years

Partnerships WHO, GIZ, UNFPA, KfW and the WB

For the purpose of the evaluation based on the Results framework of the Phase 2, the ET developed the Theory of Change (TOC) presented on the Figure 7 below. The overall objective of the Project as stipulated in the project description will be achieved if:

- MoH institutional capacity is enhanced to lead and facilitate MCH health sector reform: o Enabling environment is established facilitating effective MCH policy implementation; o MoH is empowered to effectively coordinate and facilitate evidence based MCH

policy implementation; o The MCH care facility certification system is established and promotes continuous

quality improvement interventions; - MCH health workforce capacity is built in modern MCH technologies at all levels of MCH

service delivery system and

14

- Population’s health seeking behavior is improved.

In support of these preconditions, the IMCHS (Phase 2) project applies different strategic approaches.

- Strengthening of the continuum of care by focusing on preventive, promotive, and curative MCH services on preconception, pregnancy, childbirth, postnatal and early childhood period. All levels of health system needs to be resilient, effective and efficient for providing continuum of care so that every child will have an equal opportunity for thriving with quality medical care available in the country.

- Institutionalization of informed decision making capacities of the health system following internationally accepted norms and practices will lead not only to improved quality of health care services (antenatal, postnatal and childhood), but would also strengthen the referral system.

- Strengthening community based health care system - This needs to be integrated within community based health care system through imparting better parenting skills and empowerment of families to demand and utilize health care services in timely manner.

All this will lead to better health outcome, strengthening referral system, improved nutrition and demanding the accountable and responsive health care system that contributes to achieve the overall objective.

Figure 7: Project Theory of Change

While strategies applied by the project are adequate to meet stated impact and outcomes, attainment of objectives is exposed to the following risks:

- Government’s ability to prioritize health spending and particularly ensure allocation of adequate financial resources for the delivery of quality MCH services in given economic constraints;

ContributetohumandevelopmentinUzbekistanwithaspecialemphasisonMotherandChildHealthCare

(MDGs4and5).

ThequalityofMaternalandChildHealthservicesimprovedatalllevelsofMCHservicedelivery

RESULT1:Theeffec venessofMCHreformimplementa onimprovedbyenhancedins tu onalcapacity

RESULT2:MCHservicequalityimprovedthrougheffec veuseofMCHmoderntechnologiesatalllevelsofMCHservicedeliveryandimprovedchildcareandhealth-seekingprac ces

• SupporttheMoHtoestablishandoperatetheMCHCoordina onCommi ee

• EstablishMCHcarefacilityformalcer fica onsystemandprocess

• Elabora onandpilo ngofqualityimprovementmechanisms

forMCHcarefacili es• Pilothealthsystemstrengtheningac vi esintwopilotregions• Performneedsassessmentanddevelopmentofspecifica ons

foressen almedicalequipmentforprovisionofqualityEPC,NR&ENBCservices

• Conducttrainingandsupervisionofhealthcareprovidersonmaternal,newbornandchildsurvivalpackagesaspertheapprovedstandardprotocolsandguidelines

• SupporttheMoHtoimplementthenewbornsurvival,childsurvival,andnutri onmodulesingraduateandpostgraduatecurriculafor

MCHcareprofessionals• TrainhealthcaresystemmanagerstosupportimprovementofMCH

careservices• Establishapla ormfordiscussionandexchangeofexperienceand

dissemina onofthebestprac cesonMCHcaresectorreformissues• Establishcommunitybasedbehaviorchangemechanismbasedon

par cipatorylearningac onapproach(PLA)undertheownershipofselectedMahallasandprimaryhealthcareins tu ons(SVP).

• DevelopPLAmodules,BCCmaterialsandtoolswhichpromotehealthybehaviors

• TrainpatronagenursesandMahallaAdvisorsfromWomen’s

Commi eestoactasfacilitatorsinimplementa onofcommunityac vi es

TechnicalExper seFinancialResourcesandresourceleveraging

Equipmentin-kind Evidencebasedpolicydialogue

IMPACT

OUTC

OME

OUTPUT

ACTIVITIES

INPUTS

15

- Pursued leadership at national, sub-national and local levels, including at the facility level for institutionalization and promotion of continuous MCH service quality improvement initiatives;

- Ability to ensure uninterrupted supply of resources (human, financial, and material resources);

- Continued engagement of local communities for community empowerment initiatives.

1.7 KEY PROJECT STAKEHOLDERS

The main international development partners, their role in health care system reform in Uzbekistan, their respective contributions, and collaboration with the project is described below: - The World Health Organization (WHO): The WHO Country office provides technical

support to the IMCHS Project Phase II at national level, and in implementation of the health system strengthening of provincial and district health departments in two pilot regions (Namangan and Republic of Karakalpakstan (RoK)).

- The Deutsche Gesellschaft fur Internationale Zusammenarbeit (GIZ): The project has collaborated with GIZ through joint activities on quality improvement, piloting of certification of MCH care institutions and Behavior Change Communication (BCC) community activities in Republic of Karakalpakstan.

- The Kreditanstalt fur Wiederaufbau (KfW): KfW has cooperated with the project in implementation of the Joint Monitoring Team (JMT) visits. The project has agreed with KfW to cover the H-IMCI training activities in specialized pediatric care, and for the development of related clinical protocols in pilot oblasts. The project provided trainings on use of Neonatal Equipment (NET) to 14 Children Oblast Hospitals equipped by KfW.

- The World Bank (WB): The project collaborated with the WB Health 3 project through joint implementation of IMCI training activities for doctors and nurses at the Oblast level. It was agreed that the WB Health 3 project will implement the H-IMCI training activities in the pilot Oblasts of the IMCHS Phase I Project, and will train the nurses from the Oblast Children’s Multi-profile Medical Centers in all Oblasts of Uzbekistan.

- The United Nations Population Fund (UNFPA): has cooperated with the project in implementation of JMT and Near Missed Cases Review (NMCR) approach. The project and UNFPA have agreed to implement their activities in different regions, to prevent duplication and overlapping. The project agreed with UNFPA to cover the Effective Perinatal Care (EPC) training activities. UNFPA was responsible for the implementation of Emergency Obstetrics Care (EMOC) training activities and assessment of national perinatal centers in terms of readiness for EMOC. In addition, the project facilitated the linkages b/w the UNFPA and LDS Charities to implement the Helping Mothers Survive training (component related to obstetric hemorrhages).

- The United Children’s Fund (UNICEF): is seen by the Government as the subject matter expert on mother and child health services and as a lead partners that support the MCH sector reform. UNICEF is working with the MoH of Uzbekistan to improve access to and quality of health care for mothers and children by working at different levels: policy, system and community. UNICEF is an implementing entity of the IMCHS Phase II project in close partnership with MoH and EU Delegation in Uzbekistan.

In December 2012, MoH established the Maternal and Child Health Coordination Council (MCHCC) to coordinate the efforts of the MoH and relevant national health institutions (Republican Specialized Scientific and Practical Medical Center of Pediatrics, Republican Perinatal Center, Institute of Health and Medical Statistics) and international development partners (EU, UNICEF, WHO, GIZ, KfW, GIZ, WB) in the area of MCH. During the implementation of the IMCHS Phase II project, this mechanism also serves as a project Steering Committee, to help position the project within the strategic framework of health sector reforms. A secretariat has been established to support MCHCC’s effective functioning and two technical working groups (TWG) are in charge of programmatic and medical education related issues.

16

2. EVALUATION PURPOSE AND METHODOLOGY 2.1 EVALUATION RATIONALE, OBJECTIVES AND SCOPE

This summative evaluation will come at a time when:

a. The IMCHS project phase II will conclude its activities, and b. The Country Program of Cooperation between the Government of Uzbekistan (GoU) and

UNICEF will be finalizing the 2010-2015 cycle and will embark onto implementation of a new 2016 – 2020 Country Program (CP).

In this context, the Evaluation offers the opportunity to critically assess IMCHS project phase II as a stand alone project as well as in the broader context of scale up cost-effective interventions related to the maternal, newborn and child health care and their contribution to maternal and child health and well-being. The main objectives of the evaluation are:

Assess the relevance, efficiency, effectiveness, sustainability, coherence and to the extent possible, impact of the project results;

Identify and document success, challenges and lessons learned;

Provide recommendations to guide: i) implementation of the next program cycle, and ii) policy level decision making by relevant stakeholders and international development agencies.

The evaluation will cover most of the 2nd phase implementation period from July 2012 to December 2015. The geographical scope will include the project targeted regions and both the national and sub-national levels. Each evaluation criterion will be analyzed from the perspective of assessing what are the project activities’ implications on:

Final beneficiaries: caregivers, families, mothers and children; Service providers: health care professionals whose capacity has been built (including

doctors, midwives, patronage nurses, health facility managers); Sub-national decision-making level: Regional health authorities and Hokymiats (local

governments) National decision-making level: national authorities and key stakeholders (Ministry of

Health, Ministry of Finance, Cabinet of Ministers) and Development Partners: WHO, UNFPA, WB, ADB, USAID, EC, GIZ).

The geographical scope of the evaluation study is 6 regions of the country where the ''Improvement of Mother and Child Health Services in Uzbekistan, Phase II'' (IMCHS Phase II) program is implemented. The evaluation period encompasses approximately three-year period from the program launch in 2012 up to September 2015.

2.2 TARGET AUDIENCE FOR THE EVALUATION REPORT

The findings of the evaluation will be used as a basis for advocacy and planning between UNICEF and national actors, as well as by international entities. The Government of Uzbekistan and UNICEF to inform the strategies to be applied in the new Country Program 2016-2020 will utilize the knowledge generated by the evaluation.

- The findings and recommendations will primarily be addressed with policy makers and project managers, both internally in UNICEF and externally in government, partner organizations and academia.

- A better documentation of results achieved and identification of most effective strategies and interventions should also contribute to mobilizing additional funding for achieving further reduction of avoidable child deaths in Uzbekistan.

- The evaluation will also document lessons learned, which will contribute to formulating policies to support further progress in reducing infant and child mortality and morbidity.

17

Specifically the evaluation findings and recommendations will:

- Aid the MoH to review the current maternal and child health policies and develop/revise a comprehensive and evidence-based MCH reform strategy. This assessment will help to identify the gaps and bottlenecks as well as results of the analysis will be used for better planning for MCH sector performance.

- UNICEF: To evaluate partnership and intervention as a whole and inform their adjustment for better results in the MCH sector with an equitable approach. Furthermore, the findings of evaluation will inform UNICEF’s advocacy and project strategies and guide their assistance to the MoH in the development of an evidence-based reform process.

- WHO, UNFPA, GIZ, WB, KfW, EU and other partners: The knowledge generated by the evaluation will also be used by international partners to review their partnership around the MCH sector and adjust accordingly for the achievement of the organizational and national MCH policy targets; review and evaluate support strategies to the MCH sector reform in the country.

2.3 GUIDING PRINCIPLES AND OVERALL APPROACH

The evaluation will be fully guided by the UNICEF Evaluation Policy and its guiding principles. In addition the evaluation team (ET) will adhere to the following principles:

- Custom tailored and built on existing knowledge within UNICEF and in the country; - Participatory and inclusive - ensuring participation of all involved and appropriate

stakeholders and taking into account diverse viewpoints; - Integrity and honesty in reporting strengths, weaknesses, successes and failures of the

program design and implementation using robust evidence; The mixed method approach, which combines the qualitative and quantitative components described later in the document, will be used to achieve the evaluation objectives and to respond to the specific evaluation questions as specified in the TOR. 2.4 EVALUATION CRITERIA AND FRAMEWORK

The evaluation will be summative in nature. It will provide a summative assessment of the IMCHS project in Uzbekistan at the end of UNICEF support – the extent to which the project was relevant to the needs of mother and children in Uzbekistan, has had impact, and is sustainable. The evaluation will also examine project’s support, judging how effective and efficient the support was and to what extent it contributed to increased equity in services provided.

As stated in the Terms of Reference, the evaluation will examine the impact (to the extent possible), relevance, effectiveness, efficiency, sustainability 44 , and coherence of the project’s contribution along the MORES framework. For this purpose, the evaluation will utilize OECD DAC evaluation approach 45 as defined in the UNICEF guidance on equity-focused evaluations, 2011, though it was revised/adapted according to the specific evaluation questions per each criterion requested in the TOR. The overall evaluation approach is based on the theory of change spelled out in the results framework and detailed in the project inception report (Figure

7 on page 14).

44 Evaluation Criteria 45 The DAC Principles for the Evaluation of Development Assistance, OECD

18

For achieving evaluation objectives, the evaluation framework (EF) (ANNEX 3: EVALUATION FRAMEWORK) has been developed. The EF structures questions as indicators, which can be measured or assessed during the evaluation. It also identifies the sources of information, methods the evaluation will apply, the range of documents it will review and key informants to interview for each question. The EF will be part of a process rather than simply an end product to ensure that there is clarity and agreement about what is required and how the evaluation structure and methodology are derived from that.

The evaluation focus areas will be assessed against these criteria and will attempt to answer questions stipulated in the TOR. The evaluation will also integrate Human Rights (HRBA) and equity (EQ) based dimensions into all evaluation criteria. The evaluation questions will be informed by i) the UNEG guidance on how to integrate Human Rights HRBA & EQ considerations in evaluations46 and ii) UNICEF’s equity based evaluation47. It will examine to what extent the IMCHS project benefited right-holders, including a wide range of project beneficiaries and strengthened the capacities of duty bearers and other key players to fulfill their obligations and responsibilities. More specifically, the EF will include additional questions under each evaluation criterion (when needed) to inform findings on HRBA&EQ related issues.

Apart from the EF the Project Results Framework (RF) will be used to demonstrate how project activities eventually results in achieving its objectives—kind of a road map that shows project’s final destination and how the project will get there. Therefore, in order to assess attainment of stated targets for outcomes and results/impact, the evaluation will collect and carefully assess data provided in the RF and or collected through secondary data analysis.

Figure 8: Types of Impact

Where possible analysis of attribution and causal relationships will be established by presenting data on the indicators set out in the logical framework of the interventions, and where possible, explaining recorded changes by the intervention(s) under the project. Finally, if any unintended consequences will be noted looking at expected and unexpected positive and negative impact and findings will be fully elaborated.

Text Box 1: Questions on lessons learned

To document lessons learned and good practices of the pilot project activities, along with evidence of outcomes, based on the findings along each of above-mentioned measures/criterion, the evaluation will draw conclusions, provide recommendations, and address broader questions on lessons learned, as shown in Text Box 1. 2.4 EVALUATION PROCESS

The evaluation will be implemented in three phases (Figure 9).

46 Integrating Human Rights and Gender Equality in Evaluation-Towards UNEG Guidance, UNEG, 2011 47 How to design and manage Equity –focused evaluations, UNICEF, 2011

Expected Positive

Unexpected Positive

Expected Negative

Unexpected Negative

- What worked well and did not work well on the project?

Or what could have been done differently, if there is a

possibility to start the project over?

- What are the key lessons learned from UNICEF’s support and the conclusion of this support?

- To what extent could UNICEF and or government utilise these lessons and experiences to inform its policy going forward?

- What are some key recommendations that can be utilized by other countries in the region?

- What are innovation and good practices?

19

PHASE 1: PREPARATION PHASE The ET conducted desk review and prepared detailed evaluation, design which includes stakeholder mapping (key informants), evaluation and results frameworks, interview and focused group discussion guides, as well as a detailed plan for data collection, including selection of project sites and beneficiaries. The methodology and instruments will be shared for validation to UNICEF CO and the Steering Group established for evaluation-representing UNICEF CO, partners and other stakeholders. Key deliverable(s):

- Draft Inception Report including: detailed evaluation methodology; data collection tools; sampling methodology and list of selected oblasts/districts/facilities; list of key Informants for interviews and focus group discussions; implementation timelines and milestones.

Figure 9: Phases of Evaluation

PHASE 2: DATA COLLECTION/FIELD PHASE A mission of around ten days to the country will be undertaken and all data collection exercises (qualitative as well as secondary quantitative) will be completed. At the end of the evaluation mission, the consultant will present preliminary findings and recommendations to the key stakeholders for validation and collect initial comments. Key deliverables: - Power Point Presentation on preliminary findings, lessons learned and recommendations PHASE 3: REPORTING PHASE In this phase the consultant will prepare the draft evaluation report. The draft report will be subject to a formal review process by stakeholders and UNICEF. The final report will incorporate recommendations and comments by the reviewers. Key deliverables: - Final Evaluation Report

• Desk Review

• Evaluation methodology and tools

• Data collection schedule

• Inception Report

PHASE 1: PREPARATION

• Data collection

• Preliminary analysis

• Presentation of preliminary findings and recommendations

PHASE 2

FIELD PHASE • Triangulation of data and analysis

• Draft Report preparation

• Solicitation of stakeholeders'comments

• Submission of final report

PHASE 3

REPORTING PHASE

20

2.5: PROPOSED ASSESSMENT SCHEDULE AND EVALUATION TEAM MEMBER RESPONSIBILITIES

2.5.1 PROPOSED ASSESSMENT SCHEDULE

Table 3: Proposed Assignment Schedule

ACTIVITY DURATION LOCATION DELIVERABLES

1 Desk Review 31.08.2015 – 10.09.2015

Home based Documents shared with evaluator for analysis

2 Preliminary discussions with the Evaluation Management Team and other stakeholders

7.09.2015 Home based Preliminary discussions with Evaluation Management Team carried out to obtain in-depth common understanding of the conceptual framework

3 Preparation of Evaluation Inception Report

5.09.2015 – 16.09.2015

Home based Inception report containing evaluation rationale, objectives, design, methodology, tools, limitations, task distribution, schedule, list of stakeholders, etc.) Submitted to UNIEF CO

4 Incorporation of comments into the Inception report

20.09.2015 –

28.09.2015

Home based The Final Inception Report incorporating provided comments is submitted to UNICEF

5 Logistical Arrangements 1.09.2015 – 17.09.2015

Home based Ticket purchased, Stakeholder meetings arranged, in-country travel arrangements completed, hotel booked)

PHASE II - FIELD

4 Data Collection 2.10.2015- 14.10.2015

Uzbekistan Field visits, Meetings, FGDS, Data analysis

6 De-briefing Uzbekistan Presentation of initial Evaluation findings and Recommendations, Findings validated and initial comments solicited

PHASE III - REPORTING

7 Preparation of Draft Evaluation Report

16.10.2015 – 4.11.2015

Home based Draft Final Report submitted for comments

8 Comments from UNICEF 5.11.2015 – 11.11.2015

Home based Comments on draft report provided

9 Comments incorporated in the Final Report

12.11.2015 –

18.11.2015

Home based Final Evaluation Report submitted to UNICEF CO

2.5.2 EVALUATION TEAM MEMBERS’ RESPONSIBILITIES

The composition, roles and responsibilities of the evaluation Team members are described in Table 4 below.

Table 4: Evaluation Team composition, Level of Effort and responsibilities

POSITION NAME LOE RESPONSIBILITIES

Team Leader & Public Health Expert, Evaluation specialist

IVDITY CHIKOVANI, MD., PhD

40 Evaluation Team Leader responsible for overall design, planning, implementation and preparation of the final evaluation report.

Serves as a Public Health Specialist on the Evaluation team. Takes lead in formulation of evaluation framework, method and tools as well as format of the final deliverables, performs desk review, data collection in the field, triangulation and produces required inputs for the final report.

Co-evaluator and MCH Specialist

MAYA KHERKHEULIDZE, MD, PhD

25 Contributes to the design of evaluation methodology and tools; Performs desk review with the focus on MCH issues, collects data in the field, performs data triangulation and produces required inputs for the final report.

Co-evaluator and Child rights Specialist

KONSTANTIV OSIPOV, MD., MA

10 Contributes to the design of evaluation methodology and tools, performs desk review with the focus on Human/Child Rights Issues, data triangulation and produces required inputs for the final report.

Project Coordinator

MAIA UCHANEISHVILI

7 Contracting of evaluation consultants (national and international); Supervision and monitoring of the Evaluation implementation, organization of logistical support; liaison with UNICEF CO; collection of documents for desk review and solicitation of counterpart comments; organization and logistics; printing of final report and periodic reporting to UNICEF CO.

Field Coordinator (National)

SHUKHRAT ABDULLAEV

15 Responsible for provision of logistical support in the field to the Evaluation Team, collection of documentation for desk review; organization of country filed work logistical support; Liaison with UNICEF CO and provision of other logistical and translation support as requested.

21

2.5.3 EXPECTED ROLES AND RESPONSIBILITIES FROM THE COMMISSIONING ORGANIZATION AND OVERSIGHT COMMITTEE

The UNICEF CO Evaluation Management Team (EMT) will oversee the management of the evaluation process, liaison between the evaluator(s) and partners / stakeholders involved, ensure quality of the report and determine the management's response to the evaluation findings and recommendations. It will coordinate with key stakeholders through MCHCC at key milestones such as inception and data collection stage, report validation and discussion of findings and recommendations. It will also ensure operational support as required, including support in primary data collection where needed to complement what available from the existing monitoring systems and other documents. The EMT will also ensure: - Provision of logistical support for meeting arrangements in Tashkent and pilot regions to be

visited by the Evaluation Team; - Provision of feedback on draft inception and final evaluation report within 5 working days; - Grant small office space for the evaluation team during the field data collection phase,

whereas the evaluation team will use own computers for work; - Support for in-country transportation for field visits and for meetings (as indicated in the

TOR); - Will take entire responsibility for organization of Stakeholder Validation Workshop

organization, and to assume responsibility for covering travel, per-diem and logging costs of participants when needed.

2.6 METHODS OF DATA COLLECTION AND ANALYSIS

2.6.1 METHODS OF DATA COLLECTION

The methodology will comprise a mix of site visits and observations, face-to-face semi structured interviews, focus group discussions, desk-based research and review of existing reports, documents and available secondary data. All data collected during the evaluation will be analyzed using NVivo 10™ software48. Summary of Methods and data collection framework are outlined below (Figure 10): Figure 10: Data Collection Framework

Desk Review (DR): Review of documents was a major part of the assignment. The ET consulted with and obtained from UNICEF all necessary documents. The list of documents reviewed is provided (ANNEX 1: PRELIMINARY LIST OF DOCUMENTS FOR DESK REVIEW). The desk review also studied qualitative and quantitative secondary data available around the themes of the evaluation and informed preparation of data collection tools.

48 NVivo is a qualitative data analysis (QDA) computer software package produced by QSR International. It has been designed for qualitative researchers working with very rich text-based and/or multimedia information, where deep levels of analysis on small or large volumes of data are required. The software allows users to classify, sort and arrange information; examine relationships in the data; and combine analysis with linking, shaping, searching and modeling.

DESKREVIEW

INTERVIEWS,INDIVIDUALGROUP

DISCUSSIONS

FOCUSGROUPDISCUSSIONS

TRIANGULATION&VALIDATION

SITEVISITS&OBSERVATIONS

22

Site Visits (SV): The programme coverage extends to 6 Regions. For the evaluation purpose, 20% of supported regions (3 regions) were sampled using the multistage sampling methodology. Parameters used for sampling comprises geographical location, births rates in project target regions as well as Infant and U5 Mortality Rates.

Table 5: Site Sampling

Geographical Location

Geographical Cluster

Birth rate Sub-cluster IMR Sub-cluster

City Tashkent Capital 0 M 2 H 3

Bukhara Region Southwest 1 M 2 L 1

Tashkent Region Northeast 2 L 1 M 2

Fergana Region East 3 H 3 M 2

Khorezm Region Northwest 4 H 3 M 2

Republic of Karakalpakstan Northwest 4 H 3 H 3

Legend: H – High, M-Medium, L-Low Shaded are sampled regions

In the first stage all regions were listed by their geographical location and clustered representing different parts of the country (Table 5). On the next stage regions from each geographical cluster were mapped against Birth Rates and IMR49. As a result the regions provided in Table 5 were selected for site visits (Karakalpakstan, Tashkent and Bukhara). Final sampled project regions represent different geographical location, regions with high, medium and low birth rates and high, medium and low Infant Mortality Rates.

The ET will carry out In-depth Interviews with local key stakeholders including Community Women’s Committees as well as visit service provider facilities (maternity and pediatric wards, PHC facility) and training facilities where applicable in selected locations. In-depth Interviews (IDI): IDIs with various key stakeholders and individuals will be an important source of evidence for many of the evaluation questions. The objectives of IDI’s are twofold: i) solicit stakeholder’s views on the key evaluation questions and ii) gather data and other evidence that supports analysis.

Although most of key informants may provide information relevant to both objectives, the evaluation team categorized IDIs as follows:

Top-level interviews - to be conducted with senior representatives of stakeholders. For these interviews the evaluation team will focus particularly on questions related to policy content and its relevance to achieving ownership and sustainability; policy impact on national processes, implementation issues, challenges and plans etc.

Subject-specific interviews – to be conducted with officials/representatives of the ministry of health, national and local government representatives, facility managers, staff providing services to pregnant women, young mothers and children, etc. and interview will focus on particular aspects of the evaluation such as intended and unintended consequences, strength and weaknesses, opportunities etc.

Facts finding/data interviews – In addition to the above types of IDIs the evaluation team will access valuable sources of “interview based evidence” on more detailed or specific points/issues.

Prior to visiting key informants (ANNEX 2: PRELIMINARY STAKEHOLDER LIST), IDI interview topic guides will be developed based on the Evaluation Framework to help ensure systematic

49 Due to the absence of data on U5MR, this indicator was not used for sampling purposes

23

coverage of questions and issues. The interview topics are selected around the evaluation questions, but grouped and targeted according to the organization and/or individual to be interviewed (ANNEX 4: IDI GUIDE). Focus Group Discussions (FGD): Despite sensitivities towards carrying out FGDs due to country-specific social norms expressed in the TOR, the ET selected FDGs as another method for data collection as this method is particularly suited for obtaining several perspectives from providers and beneficiaries about the improvements in service quality. Based on the discussions and mutual agreement with UNICEF CO, in the absence of formal ethical review mechanism in the country, the Ethical Review Panel has been established by UNICEF CO to clear FGDs with direct beneficiaries.

In the context of this evaluation, the FGDs serve to capture the perspectives of service providers, as well as of direct beneficiaries. Focus group discussions will be organized in pilot project sites for two groups of stakeholders and beneficiaries: i) Service providers and ii) Beneficiaries.

FGDs with service providers will include staff of project targeted health institutions, which will be visited during the evaluation field data collection phase. The topics that will be pursued with service providers include but will not be limited to: the relevance of the training content, the day to day effectiveness of the knowledge and skills gained through project supported trainings, and broader topics such as the effectiveness and sustainability of service delivery, etc.

FGDs with beneficiaries will bring together a balanced mix of pregnant women and young mothers. The purpose of FGDs with beneficiaries is to gauge the extent to which project support might have contributed to improved demand of high quality services and healthy behaviors, as well as utilization of services, measured by satisfaction. Furthermore, FGDs will also attempt to identify key bottlenecks/challenges and unmet needs of the target population. The principal topics to pursue as part of these FGDs are: i) degree of accesses to services; ii) their perception on the service quality; iii) bottlenecks, challenges; iv) unmet needs.

FGDs participants will be recruited through each in-patient facility to be visited by the ET. Each FGD will target eight to ten participants and will last about 60 minutes. For each FDG the FGD guides are designed (ANNEX 6: FGD GUIDES). The ET will conduct one FGD per each group in each selected site, in total 6 FGDs in both selected project sites. The entire discussion will be tape-recorded and afterwards transcribed in verbatim. 2.6.2 DATA SOURCES The four major sources of data will be used during the Evaluation.

People - Individuals will be consulted through individual (IDI) interviews and focus groups;

SITE visits - Data collected during the visits to sampled project supported sites.

Documents - All project and thematic area related documents (primary and secondary data sources) would be reviewed. Detailed list of documents to be consulted are provided in ANNEX 1: LIST OF DOCUMENTS REVIEWED.

Quantitative analysis - The ET will utilize quantitative analysis to examine changes in selected but comparable indicators from available data (quantitative statistics, monitoring data, researches and studies, etc.).

24

2.6.3 METHODS OF DATA ANALYSIS

Triangulation of Findings: Both quantitative and qualitative data will be analyzed to assess evaluation domains and criteria. Findings based on qualitative data will be triangulated across key informants, compared with available documentary evidence and validated in the focus groups before drawing conclusions and formulating recommendations. - Qualitative data analysis will entail documentation, conceptualization, coding, and

categorizing, as well as examining relationships. More specifically, it is expected that the qualitative data will allow obtaining in depth perspective on context, actors and processes related to the programs’ design and implementation and testing/identification of the factors shaping the pattern of UNICEF contribution. A framework analysis approach will be mainly used for the analysis of the qualitative data obtained through the variety of the data collection methods described above. This approach is sought to allow capturing the complex environment and wide range of new issues and propositions that may emerge during the evaluation process, rather than focusing analysis on solely on predetermined propositions and prior understandings, as required in a purely deductive approach.

- Quantitative data analysis will be made based on available secondary data (national

Statistics, Global databases, Research, surveys and studies) and in comparison with original project objectives.

Data Verification – The ET will review data from various sources to answer main questions of the evaluation. Responses from each data source will be compared in order to identify discrepancies in country data. For treating response variations the team will establish a protocol for “treating discrepancies in the data”. Information derived from each of the sources of qualitative and quantitative used at every stage of the study will be triangulated within and between data sets with the aim of identifying common understandings of the experiences of issues at focus, as well as differences of opinion between various stakeholders. Following triangulation, the data sets will then be used to develop specific analyses, such as timelines summarizing the chronology of program implementation, descriptions of particular processes used in the design or implementation of the programs and stakeholder analyses of actor positions on specific features of the design and implementation at specific time. The methodology and data collection tools might be further revised based on the feedback by Ethics Review panel. Appreciative enquiry: An approach that will seek to explore successes and positive experiences in a dialogue with individuals and groups of people and will be applied in order to strengthen understanding of why something worked well and why some did not, and how success can be replicated or mistakes avoided. ET will have daily discussions on the information collected each day – to validate findings but also to drive new areas of enquiry in an iterative way. 2.6.4 DATA QUALITY ASSURACE The following techniques will be used during the evaluation to assure the quality: - Elements of multiple coding, with regular cross checks of coding strategies interpretation of

data between local and international experts participating in the study and this will represent one of core activities of the regular meetings and/or online conferences during the evaluation when the data is collected through in depth interviews and focus group discussions;

- Respondent validation, which will involve cross checking interim and final evaluation findings with key informant respondents, along with proposed mode of work with key stakeholders on

25

relatively continuous basis are expected to enhance the rigor of the proposed evaluation and the evaluation results

- Triangulation of data collected from different sources during the evaluations, may help to addresses the issue of internal validity by using more than one method of data collection to answer proposed evaluation questions;

Table 6: Robustness Ranking for Evaluation Findings

However, it is expected that the quality of the data will vary. As such, to account for the data quality and assess the strength of our conclusions we intend to use the “robustness scoring” approach for each finding. Consequently, four scores (A to D) will be used in this process. Assignment of the score will depend on an assessment of the combination of the following two criteria: a) the extent to which qualitative and/or

quantitative evidence generated from different sources point to the same conclusion and b) what is the quality of the individual data and/or source of evidence (e.g., as determined by sample size, reliability/ completeness of data, etc.). Table 6 shows detailed description for “robustness score” assignment. 2.6.5 EVALUATION LIMITATIONS The evaluation may face number of limitations: Table 7: Evaluation limitations and mitigation measures

LIMITATIONS

PROBABILITY/ LIKELIHOOD

(Low, Medium, High)

IMPACT (Minor/

Moderate Major)

METHODS EMPLOYED TO OVERCOME LIMITATIONS

Availability and or access to quantitative data, alongside with data completeness or data accepted as authoritative or at least a “best estimate” or value will potentially influence the quality of the evaluation findings and raises risks of impact /outcome evaluability.

High Major Hence, the national statistics is considered not to reflect accurate data, the evaluation will use Global Health databases and studies (DHS, HUES, MICS) where applicable and examine potential impact of the project based on latest available data. However in most cases available data is outdated (as of 2013).

Measuring impact right at the end of the project is likely not to allow for assessing project impact

High Major To mitigate this limitation, the ET will attempt to measure observable results with the envisaged pathways of change as per TOC and examine possible contribution of the project to the impact.

Some documents studied during the “Desk Review” indicate that they are only for internal use. This will limit the evaluation to use respective information and make reference to the

Medium Moderate The ET will seek UNICEF’s guidance on this matter.

RANKING DESCRIPTION

A The finding is consistently supported by the full range of evidence sources, including quantitative analysis and qualitative evidence (i.e., there is very good triangulation); and/ or the evidence source(s) is/are of relatively high quality and reliable to draw a conclusion (e.g., there are no major data quality or reliability issues).

B There is a good degree of triangulation across evidence, but there is less or ‘less good’ quality evidence available. Alternatively, there is limited triangulation and not very good quality evidence, but at least two different sources of evidence are present.

C Limited triangulation, and/ or only one evidence source that is not regarded as being of a good quality.

D There is no triangulation and/ or evidence is limited to a single source and is relatively weak; or the quality of supporting data/ information for that evidence source is incomplete or unreliable.

26

source document.

The Degree of project financial data disaggregation may potentially limit assessment of project efficiency.

Medium Moderate The ET requested UNICEF to furnish project financial data and will seek UNICEF’s Financial Department staff support to disaggregate data in a way that could be analysed to draw project efficiency conclusions.

Possible unwillingness and/or inability of key stakeholders alongside with possible interview fatigue – there being no “reward” for being interviewed for interviews - may be another limitation of the evaluation.

Low Minor The Evaluation Team will use “snow ball” method to identify alternative key informants. Furthermore, in case of unavailability, the ET will try to schedule phone/Skype interviews to minimize number of non-interviewed key informants

Absence of “Ethical Committee” approval

Low Moderate Ethics review board was established at UNICEF CO which will discuss the data collection methodology particularly for FGDs.

Few number of service providers in project targeted health facilities, particularly in PHC facilities

Medium Minor In such cases the evaluation will use the Group Interview/In –depth interview method for qualitative data collection.

Due to time and budget constraints, the proposed evaluation method does not include an extensive population-based survey, but largely relies on service statistics; baseline and follow up Health Facility Assessment and KAP surveys, as well as qualitative data collected through interviews and focus group discussions. 2.7 STAKEHOLDER PARTICIPATION AND ETHICAL ISSUES

2.7.1 PARTICIPATORY APPROACH In order to develop ownership and ensure the involvement and interest of the stakeholders for sustainable changes and future developments, the evaluation will be conducted in a participatory way, involving policy makers, program staff, service providers and partners’ staff, beneficiaries and their partners and other people directly or indirectly involved in the project. The ET will ensure active participation of key stakeholders in all phases of the evaluation process. The evaluation findings and recommendations will be presented and verified at stakeholders meeting before final version of the report is produced. In close consultation with UNICEF the evaluation team will identify key stakeholders to be invited for the debriefing meeting. Comments, suggestions and clarifications provided by the stakeholders will be adequately addressed in the evaluation report. Moreover, initial draft of the report will be shared for comments and feedback received will be reflected in the final report. 2.7.2 ASSURANCE OF INDEPENDENCE AND IMPARTIALITY During the evaluation process consultant will ensure impartiality and independence at all stages of the evaluation process, which will contribute to the credibility of evaluation and the avoidance of bias in findings, analyses and conclusions. Furthermore, consultant will ensure a maximum level of objectivity. Statement of facts will be methodically clearly distinguished from assessments; the different perspectives will be taken into account, as well as strengths and weaknesses; results, conclusions and recommendations will be supported by evidence and will be comprehensible. To guarantee reliability of the evaluation findings, consultant will utilize all available data in order to prove the assessment and the conclusions in a credible fashion .

27

2.7.3 ETHICAL ISSUES

While designing the evaluation methodology, the UNEG ethical guidelines for evaluation50 was consulted and the following approaches will be utilized: - The ET will try to keep evaluation procedures (FGD and IDIs) as brief and convenient as

possible to minimize disruptions in respondents work process; - To ensure that potential participants can make informed decision ET will obtain consent;

inform about the purpose of evaluation and final outcome; explain the process and duration of interview and/or FGD.

- The evaluation team will also ensure respondents about the confidentiality of the source for obtained information and allow them to retain from answering the questions posed in case they feel uncomfortable to respond;

- Key informants will be interviewed face to face without presence of other individuals and their identities will not be revealed and or statements attributed to a source.

- Information will be analyzed and findings reported accurately and impartially. - UNICEF’s “Ethics Review Board” approval will be obtained prior to the field phase.

50 UN Evaluation Group Ethical Guidelines for evaluation, March 2008 http://www.unevaluation.org/ethicalguidelines

28

3: PRELIMINARY OUTLINE OF THE REPORT

The Evaluation Report will comply with UNICEF Evaluation Report Standards as outlined at http://www.unicef.org/evaldatabase/files/UNICEF_Eval_Report_Standards.pdf. Proposed evaluation report outline is provided below.

REPORT OUTLINE ACKNOWLEDGEMENTS ABBREVIATIONS TABLE OF CONTENT TABLE OF FIGURES AND TABLES EXECUTIVE SUMMARY (5 - 8 Pages) 1. CHAPTER 1: INTRODUCTION

1.1 Country Background 1.2 Socio-economic Context and Demographic Trends 1.3 Maternal and Child Health 1.4 Project Description 1.5 Key Project Stakeholders

2. CHAPTER 2: EVALUATION RATIONALE, OBJECTIVES AND METHODOLOGY 2.1 Rationale 2.2 Evaluation Objective 2.3 Evaluation scope, geographical coverage 2.4 Evaluation Framework and Criteria 2.5 Evaluation Methods 2.6 Data Analysis Quality Assurance 2.7 Stakeholder Participation and Ethical issues 2.8 Evaluation Limitations and Mitigation Measures

CHAPTER 3: ASSESSMENT FINDINGS 3.1 Relevance 3.2 Effectiveness 3.3 Efficiency 3.5 Impact 3.6 Sustainability 3.7 Coherence 3.8 Human Rights Based Approach and Gender Equality 3.9 Summary of main findings

CHAPTER 4: LESSONS LEARNED and GOOD PRACTICES CHAPTER 5: RECOMMENDATIONS 5.1 General Recommendations 5.2 Specific Recommendations ANNEXES:

Annex 1: List of Documents Reviewed Annex 2: List of Key Informants Annex 3: Evaluation Framework Annex 4: Results Framework Annex 5: In depth Interview Guides Annex 6: Focus Group Discussion Guide Annex 7: Assessment ToR Annex 8: Evaluation Team Composition and Expertise Annex 9: Other (as deemed necessary)

29

ANNEXES

ANNEX 1: LIST OF DOCUMENTS REVIEWED

1. Uzbekistan: Woman and Child Health Development Project, 2012

2. Formative Evaluation of Improvement of Mother and Child Health Services in Uzbekistan, UNICEF, 2011

3. UNICEF Annual Report 2012 for Uzbekistan, CEE/CIS, UNICEF, 2012

4. Uzbekistan profile of the Sexual and Reproductive Health (SRH) services available at PHC level, F. Kozacoka, Geneva Foundation for Medical Education and Research, 2011

5. The roles and influence of grandmothers and men- EVIDENCE SUPPORTING A FAMILY-FOCUSED APPROACH TO OPTIMAL INFANT AND YOUNG CHILD NUTRITION, USAID

6. SELF-ASSESSMENT OF PUBLIC HEALTH SERVICES IN THE REPUBLIC OF UZBEKISTAN Technical report, WHO, MoH, 2011

7. Public health risk assessment and interventions in Kyrgyzstan and Uzbekistan, WHO, 2010

8. Primary Health Care in Uzbekistan, WHO, 2008

9. Multidimensional Poverty Index (MPI) At a Glance, OPHI, 2011

10. Making Pregnancy Safer, Multi-Country review meeting on maternal mortality and morbidity audit “Beyond the Numbers”, WHO, 2010

11. Multiple Indicator Cluster Survey, 2006, UNICEF, 2006

12. Infant and Young Child Feeding Programme Review, UNICEF, 2009

13. Improving Hospital Care for Children, WHO, 2010

14. Five barriers to physician workforce development in Uzbekistan. Zukhra Karimova and Gary L. Filerman, Eurohealth Vol 16 No 3

15. Development of national strategies – case studies from five countries, European Strategy for Child and Adolescent Health and Development, WHO, 2008

16. Determinants of neonatal and under-three mortality in Central Asian countries: Kyrgyzstan,Kazakhstan and Uzbekistan, GMS Medizinische Informatik, Biometrie und Epidemiologie ISSN 1860-9171

17. Country programme document 2010-2015, UNICEF, 2010

18. Assessing Development Strategies to Achieve the MDGs in The Republic of Uzbekistan, UNDP, 2011

19. Uzbekistan Statistical Yearbooks, RoU, 2000-2013

20. European Observatory on Health Systems and Policies, Uzbekistan: Health System Review, WHO, 2014

21. MCH State Program 2009-2013, MoH, 2009

22. MDG Reports, Center for Economic Research, 2006 and 2015

23. Constitution of the Republic of Uzbekistan, Article 40. The Constitution is available (in English), http://www.gov.uz/en/constitution

24. Shokhrukh Mirzo Jalilov, Thomas M. DeSutter and Jay A. Leitch, Impact of Rogun Dam on Downstream Uzbekistan Agriculture, International Journal of Water Resources and Environmental Engineering Vol. 3(8), pp. 161-166, September 2011

25. World Bank, Republic of Uzbekistan Assessment of the Primary Health Care Reform: Transparency, Accountability and Efficiency, 20 May 2009

26. United Nations, CRC/C/UZB/CO 34 Concluding observations on the combined third and fourth periodic reports of Uzbekistan, adopted by the Committee at its sixty third session (27 May -14 June 2013) at http://www2.ohchr.org/english/bodies/crc/docs/co/CRCCUZBCO34.pdf

30

27. World Bank, Uzbekistan Living Standards Assessment, 2003, https://openknowledge.worldbank.org/bitstream/handle/10986/14759/259230UZ.txt?sequence=2

28. World Bank, Republic of Uzbekistan Assessment of Primary Health Care Reform: Transparency, Accountability and Efficiency, 2009, https://openknowledge.worldbank.org/bitstream/handle/10986/3065/445300ESW0UZ0P1LIC0Disclosed0718191.pdf?sequence=1

29. UNICEF, Right At Birth: Birth Registration in Central and Eastern Europe & the Commonwealth of Independent States, 2008,

30. UNICEF, Global Study on Child Poverty and Disparities: Field Survey Report, 2009

31. Giorgio Tamburlini, CQI in MNC care in Uzbekistan: Report of reassessment of 4 Regional Maternity Centers, UNICEF, 12 May 2011

32. UNICEF, Infant and Young Child Feeding Programme Review Case Study: Uzbekistan, 2009

33. IMCHS Project Progress Reports, 2013, 2014, 2015

34. MCH State Program 2014-2018

35. Common Country Assessment, UN, 2014

36. UNDAF 2010-2015 and 2016-2020

37. UNICEF CO Annual Reports, 2010, 2011, 2012, 2013, 2014

38. UNICEF Country Program Evaluation 2014

39. UNICEF Country Program MTR Report, UNICEF, 2013

40. IMCHS Phase 2 Inception Report, 2012 41. IMCHS Phase 2 Baseline Assessment 42. Stelian Hodorogea, Assessment Of The Quality Of Hospital Care For Mothers And

Newborns In Three Tashkent City Maternities Using Who Standartized Tool, UNICEF, 2014

31

ANNEX 2: PRELIMINARY STAKEHOLDER LIST

Mrs. Idvity Chikovani Mrs. Maia Kherkheulidze

Friday, October 2, 2015

14:00 – 15:00 (in UNICEF) Meeting with Robert Fuderich (UNICEF Representative)

Berina Arslanagic Ibisevic (UNICEF Deputy Representative)

15:10-16:00 (in UNICEF) Meeting IMCHS II Evaluation Management team: Clarify Expectations

(Svetlana Stefanet, Nargiz Shamilova, Zokir Nazarov)

16:00 – 17:00 (in UNICEF) Atul Kumar (Chief of Communication Section)

Maksim Fazlitdinov, on stretch assignment, please join by Skype! (C4D Officer) Nargiza Egamberdieva (Communication Officer)

17:10 – 18:00 (in UNICEF) Fakhriddin Nizamov for Kamola Safaeva (UNICEF Health Officer)

Svetlana Stefanet (Chief of Health Section)

Saturday, October 3, 2015

10:00-11:30 (in UNICEF) Svetlana Stefanet (Chief of Health Section) Nargiz Shamilova (Team Leader, IMCHS II) Fakhriddin Nizamov (UNICEF Health Officer)

10:00-11:30 (in Project Office) Diyora Arifdjanova (IMCHS II National Coordinator) Nasir Abdullayev (IMCHS II M&E Specialist), Review of the project training database

PLACEHOLDER 11:30 – 12:30 (in Project Office) Analysis of project training database

Monday, October 5, 2015

9:30 – 10:30 (in the MOH) Meeting with Professor L. N. Tuychiyev, Deputy Minister of Health

10:45 – 11:45 (in the MOH) Meeting with Head and Deputy Head of MCH Directorate

EU Delegation in Uzbekistan

14:30 – 15:30 Meeting with Ovidiu Mic, Head of Cooperation Section and Doniyor Kuchkarov, IMCHS II Project Manager

NCU (in the same building with UNICEF Office)

14:30 – 15:30 PLACEHOLDER for NCU. If NCU confirms: one person visits the NCU, while the other meets with the EU Delegation.

16:30 – 17: 30 (in TIPME) Meeting with Asadov D.A Head of Department, Public Health, Economics and Health Management Department by TIPME, Management in Public Health Course Director

16:30 – 17: 30 (in TIPME) Meeting with Alimova M. H, Director Medical Education Center of the Ministry of Health

Tuesday, October 6, 2016

Tashkent City Perinatal Centre

9:30 – 10:30 (Tashkent City Perinatal Centre) Meeting with Usmanova M. S., Deputy Director, NR/ENBC, Neonatal Equipment Course Director

Republican Perinatal Centre

11:00 – 12:00 Meeting with Lubchich A. S., Director, Republican Perinatal Centre Course Director: EPC, BABIES

11:00 – 12:00 Meeting with Feruza Fazilova, UNFPA National Program Officer, Reproductive Health

Republican Specialized Scientific Practical Medical Centre of Paediatrics

14:00 – 15:00 Meeting with Akhmedova D.I. Director, IMCI Course Director

15:15 – 16:15 Meeting with Deputy Director, BF/BFHI Course Director, Salikhova K.S.

32

16:30 – 17:30 Meeting with Ishniyazova N., GDM Course Director

Wednesday, October 7, 2015 (Field Visits, Bukhara Oblast)

9:15 – 10:20 Arrival to Bukhara Responsible for coordination of the meetings: Deputy MCH services, Bukhara Oblast Public Health Department

11:00 – 12:00 Meeting in the Bukhara Public Health Department Chief (Dr. Amonov I.I.) and Deputy, MCH Services (TBD)

14:00 – 16:00 Meeting with the Chief Doctor, health service providers and patients in Bukhara Oblast Perinatal Centre

16:30 – 18:00 Meeting with the Chief Doctor, health service providers and patients of Regional Paediatric Multidisciplinary Medical Centre

Thursday, October 8, 2015 (Field Visits, Bukhara Oblast)

9:30 – 10:30 Meeting with the Chief Doctor, health service providers and patients of the SVP, Kagan district, Bukhara Oblast

11:00 – 12:00 Meeting with the Chief Doctor of the District Medical Unit (DMU) of Kagan district

14:00 – 17:00 Meeting with the Heads of Pediatric and Maternity Department of Kagan DMU, health service providers and patients

20:00 Departure to airport, return to Tashkent 22:35

Friday, October 9, 2015 (Field Visits, Republic of Karakalpakstan) Departure: Mrs. Maia Kherkheulidze @ 12 pm

7:55 - 10:00 Arrival to Nukus Responsible for coordination of the meetings: Deputy Head of MCH Services, MoH RoK

10:30 – 11:30 Meeting with the Minister of Health, RoK

12:00 – 13:00 Meeting with the Chief Doctor, health care providers and patients of SVP “Juzumbog”, Kegeili district

14:30 – 17:30 Meeting with community representatives in the pilot mahalla in Chimbay region (PLA component)

18:00 Departure to airport, returning to Tashkent 19:55

Monday, October 12, 2015

WHO Country Office (CO)

9:30 – 10:30 Meeting with Asmus Hammerick, Head of WHO CO

10:30 – 12:30 Meeting with Zulfia Atadjanova, National Officer, WHO CO

14:00 – 15:30 Meeting with Mutalova Z.D. Director of Institute of Health and Medical statistics, Institute of Health and Medical Statistics

16:00 – 17:30 Meeting with the GIZ Raushan Ataniyazova, Team Leader of the Health in Central Asia/Uzbekistan regional program

Tuesday, October 13, 2015

9:30-10:30 Meeting with Leading Specialist in Nursing, Salikhodjayeva R.K (in the MoH)

11:00 – 12:00 Meeting with Dr. Nigora Karabayeva, National Coordinator, KfW/GFA

14:00 – 18:00 Consolidation of Findings

Wednesday, October 14, 2015

9:30 – 11:00 Preliminary Debriefing with the UNICEF staff

14:00 – 16:00 Debriefing meeting with the UNICEF and MOH

Departure: Thursday, October 15, 2015 Mrs. Ivdity Chikovani

33

ANNEX 3: EVALUATION FRAMEWORK

Lines in Italic are questions added by the Evaluation Team to the list of questions outlined in the TOR

n I

D

Question

Data collection methods

Type of analysis

Fe

asib

ility

Judgment and Indicators DR

IDI

FG

D

SV

Qu

alit

ative

Qu

an

tita

tive

IMPACT

Q1 Have the project activities contributed to achieving (or not) the expected impact level results?

L The project activities contributed to achieving (or not) the expected impact level results 1. Infant mortality rate 2. Neonatal mortality rate 3. Under 5 mortality rate 4. Maternal mortality rate

Quantitative judgment: decreased, increased

✪ ✪ ✪ ✪

RELEVANCE

Q2 Were the project design, strategy and approach appropriate to achieve the set objectives?

H 1. Evidence of comprehensive needs assessment/situation analysis in the Project identification/ formulation documents

2. Number and type of baseline assessments conducted 3. Extent to which the leading causes of MCH mortality and morbidity were identified 4. Evidence of local implementation capacity assessment performed Qualitative judgment: highly adequate; adequate; somewhat adequate; not adequate

✪

✪

Q3 Was the project relevant in terms of contributing to the improvement of the health and well being of mothers and children?

H 1. Extent to which the Project interventions as planned target the leading causes of MCH mortality and morbidity

2. Extent to which the Project interventions as planned target the key barriers/bottlenecks of the MCH services

3. The share of recommendations and lessons learned from IMCHS I adopted/incorporated Qualitative judgment fully address; partially address and not address

✪ ✪ ✪ ✪ ✪ ✪

EFFECTIVENESS

Q4 Has the project contributed to achieving (or not) the expected results as per Log-frame?

1. Evidence of the Project benefits delivered to the MoH (all three tiers) 2. Evidence of the Project benefits delivered to health providers (doctors and nurses) 3. Evidence of the Project benefits delivered to patients/community (including women and

men and specific vulnerable groups)

Qualitative judgment: yes, partially, no

✪ ✪ ✪ ✪ ✪ ✪

Q5 Have the project interventions been effective in facilitating MCH sector reforms with

1. Evidence of documented and disseminated Lessons Learned 2. Evidence of policy change based on the results and lessons learned from project

interventions

✪ ✪ ✪ ✪ ✪

34

respect to policy change and quality improvement?

Qualitative judgment: yes, partially, no

Q6 How effective the project interventions have been in improving service providers’ knowledge and practices in Newborn and Child Survival Packages?

1. % of health care facilities in all regions implementing updated international protocols and standards on new born and child survival

2. % of maternities in all regions implementing perinatal care in accordance with WHO standards

3. The Project benefits received as identified by health providers (doctors and nurses)

Qualitative judgment: yes, partially, no (using baseline and follow-up HFA survey results)

✪ ✪ ✪ ✪ ✪

Q7 Was the M&E system (including supportive supervision) been effective in reinforcing skills application and tracking progress?

M 1.1 Evidence of monitoring results used for the Project implementation correction 1.2 Evidence of evaluation results used for the Project implementation correction 1.3 Evidence of the JM results used for the Project implementation correction Judgment: yes, partially, no

✪ ✪ ✪ ✪

Q8 Did beneficiaries from pilot regions improve their childcare and health-seeking practices as a consequence of community based activities?

Evidence of improved child care and health seeking practices Quantitative judgment based on the baseline and end-line KAP surveys ✪ ✪ ✪

Q9 Where there any unintended positive and/or negative results and whether the negative results could have been foreseen and managed?

Evidence of unintended positive and negative results Respondents and the ET judgment ✪ ✪ ✪ ✪

EFFICIENCY

Q10 Could the intended results been achieved at a higher level of quantity / quality?

M If the plausible possibility is established, the improved results would have been achieved by: 1. Better responsiveness and flexibility of the Project management (yes/no); 2. Improved monitoring of risks and external factors (yes/no); 3. Shifting balance of responsibilities between the various stakeholders (yes/no); 4. Accompanying measures taken or to be taken by the government (yes/no).

✪ ✪ ✪

Q11 Were the available resources adequate to meet project objectives?

M Resources were adequate/non-adequate (yes, no) 1. All planned activities implemented within available budget 2. All planned activities implemented were adequately financed

Qualitative judgment: yes, partially, no

✪ ✪ ✪

SUSTAINABILITY

Q12 Do the MoH and other concerned health institutions demonstrate ownership over different project components?

H 1. Evidence of reflection of the Project supported priorities in the relevant national, sectoral policies

2. The ET judgment on the likelihood of adequate funding to become available, once the Project support ends

3. Respondents’ and the ET judgment on the likelihood of long-term maintenance of the MCH coordination mechanism.

4. Adequacy of well-trained community representatives 5. Availability of alternative funding sources for continuation of behavioral change

✪ ✪ ✪

35

interventions Respondents and the ET judgment on the level of ownership

Q13 Extent to which the Project ensured hand over of the elements/components to the Government?

1. Evidence of the handover measures of the Project elements/components Qualitative judgment: fully, partially, no hand over

✪ ✪ ✪

COHERENCE

Q14 Has the project facilitated synergies and avoided duplications with interventions and strategies promoted by other UN agencies and developing partners?

H Project facilitating synergies and avoiding duplications (Fully avoided, partially avoided, not avoided) Respondents and the ET judgment on duplication of efforts

✪ ✪ ✪

HUMAN RIGHTS BASED APPROACH

Q15 How did the project incorporate the HRBA and gender equality?

H 1. The intervention design benefitted from specific human rights and gender analyses 2. HR & GE are clearly reflected in the intervention design (log frame, indicators,

activities) Qualitative judgment no, partly and yes.

✪ ✪ ✪

Q16 Does the Monitoring systems (routine monitoring, studies and evaluations) capture HR & GE information

H 1. Evidence that monitoring systems have captured HR & GE information (e.g. the situation of different groups of people, specific indicators, etc.)

2. Progress and results reports for the intervention include HR & GE information

Qualitative judgment no, partly and yes.

✪ ✪ ✪

36

ANNEX 4: RESULTS FRAMEWORK

Indicators Means of Verification Baseline Target Progress Status

Overall Objective: Contribute to the human development in Uzbekistan with special emphasis on Mother and Child Health Care (MDGs 4 and 5)

1.1 Infant Mortality Rate - Health National Statics (MoH)

- Interagency Estimates

- 10.4 per 1.000 live births (MoH, 2011)

- 44 per 1.000 live births (Interagency Estimates, 2011)

- Less than 10.4 per 1.000 live births

- 10.1 per 1.000 live births (MoH, 2013)

44 per 1.000 live births (UNICEF, WHO, WB, UNDP, Levels and Trends in Child Mortality Report, 2011)

1.2 Neonatal Mortality Rate

- 6.3 per 1.000 live births (MoH, 2011)

- 23 per 1.000 live births (Interagency Estimates, 2011)

- Less than 6.3 per 1.000 live births

23 per 1.000 live births (UNICEF, WHO, WB, UNDP, Levels and Trends in Child Mortality Report, 2011)

1.3 Maternal Mortality Rate

- 23.1 per 100.000 live births (MoH, 2011)

- 28 per 100.000 live births (WHO, UNICEF, UNFPA, WB Trends in Maternal Mortality 2010)

- Less than 23.1 per 100.000 live births

20 per 100.000 live births (MoH, 2013) 36 per 100.000 live births (Interagency Estimate, 2013)

Specific Objective: Support the Ministry of Health to increase the quality of mother and child health services and to increase the capacity of families to make informed choices about health and nutrition.

2.1 No. of MoH normative documents on MCH care sector reforms adopted or updated according to international standards

- MoH and MoE normative documents

- Reports of IHMS, Oblast Health Departments, and MCH care institutions

- Medical Institutes curricula

- Reports of JMT visits - Report of monitoring

studies

0 10 5

2.2 Percentage of targeted MCH care institutions that apply newborn survival and child survival packages recommended by WHO and UNICEF

Maternities 13.5

Pediatric hospitals/wards 2.3%

At least 40%

25%

54%

77,7 %

2.3 Number of Medical Institutes which have

0 8 8

37

integrated at least 80% of the newborn and child survival modules in the graduate and postgraduate curricula

2.4 Family and community interventions based on PLA approach on five key healthy behaviors are institutionalized based on the results of the pilots

0 8 NA

Expected Results: Result 1. Institutional Strengthening: Skills and operational capacity of the Ministry of Health and its three tiers (primary, secondary, and tertiary) have been enhanced to effectively support the health reform process concerning the Mother and Child health services in accordance with international standards

1.1 The MCHSSC, its secretariat, and JMT are established and fully functional

- Normative documents of MoH;

- Reports of implementation phases 2013-2016;

- Reports of JMT visits; - Reports of monitoring

activities

0 At least 4 MCHSSC meetings and 4 JMT visits per year

1.2 The quality insurance mechanisms are piloted and institutionalized, based on the project results

0 QI policy developed. QI tools and methodology piloted and submitted for MoH approval

Three tools for QI assessment and certification has been developed. National team trained. SS visits in pilot facilities done. Final assessment of pilot facilities was done. Access to documents and reports to verify results is guaranteed

1.3 The database of trainers and trainees on newborn and child survival packages is developed and institutionalized

Database in place for IMCHS1 trainers for project

purposes only

Database enriched, adjusted and

institutionalized within Center of Medical

Education under MoH

Database enriched and data entered.

2 The best practices are documented and shared with all interested parties

0 At least 2 best practices publicly shared per year

Midterm result: Three (NMCR, BABIES, EMOC) best practices shared during the cross-visit to Fergana region organized for regional level health managers and management staff of perinatal centers from

38

regions of project phases I and II. Conference to disseminate best practices on hospital care for children was held on May 2015 in Bukhara city. Two workshops to disseminate best practices on outpatient care was held in July and August 2015

Result 2: Capacity development and empowerment. A comprehensive education-training process dealing with mother and child health care has been developed and put in place, at medical institutional and family level.

2.1 Percentage of MCH care system managers and relevant health professionals who have implemented the gained knowledge on the newborn and child survival packages in their regular practice

- Reports of training activities, project database of trained personnel;

- Reports of JMT visits, FUAT visits, monitoring and evaluation visits and studies;

- Normative documents of and MoH

- Reports of organized events related to discussion and exchange of experience in MCH care domain

- Reports of implementation of community activities

- Developed PLA modules and BCC materials

- Reports of monitoring and evaluation visits

a. 39.3 % of MCH care system managers from outpatient level who have implemented the gained knowledge in their regular practice.

b. 13,9 % of MCH care system managers from inpatient level who have implemented the gained knowledge in their regular practice

a. at least 60% b. at least 30%

- 59,7% of

managers/supervisors, who conducted case management observation by supervisee – (103/256)

- 43,9% of

managers/supervisors, who discussed the identified problems with the supervisee (173/256)

- 71,4% of s managers/supervisors, who studied/expounded new orders during a visit to health facilities (182/256)

2.2 Proportion of children 2 month - 5 years that received medical care by GP doctors according to approved protocols/standards

a. Proportion of children 2 m-5 years that were examined by GP doctors according to approved protocols/standards 23.1%

b. Proportion of children under 2 years of age whose caretakers are asked by GP about breastfeeding, complementary foods,

a. At least 50.0% b. 50% c. 46,5% (341/734)

a. 72,6 % b. 49,1% c. 63,3%

39

and feeding practices during this episode of illness 21%

c. Proportion of children

who do not need

antibiotic and who leave

the facility without

having received or

having been prescribed

antibiotics by GP

d. Baseline: 46,5%

(341/734)

2.3 Proportion of children 2 month-up to 5 years whose caregivers received counseling by patronage nurses according to approved protocols/standards

a. Proportion of children (2 months - 5 years) with diarrhea whose caretakers were counseled by patronage nurses (PN) to give extra fluid during illness: 55, 9 % (57/109)

b. Proportion of children 6-12 month age whose caretakers receive at list 3 recommendations on nutrition from PN: 16.9% (29/172

c. Proportion of children (2 months - 5 years), whose caretakers received at least three counseling messages from PN on when to return immediately: 12,4% (95/769)

a. 70% b. 40% c. 30%

a. 90,5 %(172/190) b. 88,8% (175/197) c. 72,4% (541/747)

2.4 No. of successfully functioning community Behavioral Change mechanisms which use the PLA approach

0 10

2.5 Percentage of families with U5 years children, who have adopted five

I) Percentage of caregivers able to recognize danger signs that require the

I. At least 50% by August 2015

II. At least 50% by August

40

key healthy behaviors according to received training

child (0-2 months old) to be seen by a doctor

: 31% II) Percentage of mothers

who breastfed their child exclusively for the first 6 months of life: 39.6%

III) Percentage of households providing for a balanced diet for children, pregnant women and lactating mothers a. % of children under

2 years of age reporting a minimum dietary diversity (% of children 6-23 months who consumed 4 or more food groups daily) : 66.5%

b. % of children fed with own produced vegetables: 26%,

2015 d. At least 75% by August

2015 e. At least 35% by August

2015

2.6 Average time parents and other caregivers in the household dedicate to the development of children under 5 years old

2 hours on average 3 hours on average

2.7 Number of caregivers who have their children’s (under 5 years old) hands washed after going to the toilet

35% on average At least 50% by August 2015

41

ANNEX 5: IDI GUIDE

SPEAK TO THE RESPONDENT:

Good morning/afternoon/evening. My name is __________________. I am a researcher carrying out a study on the evaluation of IMCHS project (Phase 2). The Main objective of the evaluation is to examine: -----------. The interview should take less than an hour. I am kindly asking for your permission if I could go ahead with this interview. All responses will be kept confidential. This means that your interview responses will only be shared with research team members and we will ensure that any information we include in our report does not identify you as the respondent. Remember, you do not have to talk about anything you do not want to and you may end the interview at any time. Therefore, I sincerely request your cooperation in responding to the following questions. However, at any time during the course of the interview, you are free to terminate the interview. Are there any questions about what I have just explained? Are you willing to participate in this interview? Yes: Proceed with questions No: Thank you. Terminate the interview. Start asking questions. Questions for IDIs for each stakeholder to be interviewed will be selected from the Evaluation Framework prior to the interview. See Table 8 below. Table 8: IDI Guide

N Question

UNICEF senior

Managment

UNICEF project

staff

MoH senior

MoH, regional local other

national stakeholders

International Dev Partner

Education System

Q1 Do you observe improvement of maternal child health indicators (mortality) in the country / oblast/district / facility since the project implementation?

X X

Q3 Please describe main achievements of the project

What were key barriers identified and how they were addressed? Please structure by (HS building blocks):

Governance & Leadership,

Service delivery

Human Resources (sufficiency,

turnover, aging, skills, workload)

Supplies (life saving equipment

(Ambu, CPAP, warmer, etc),

medications

X X X X X X

42

Information system

Financing (costing, OOP)

Demand

Q4 To what extent has the project contributed to achieving (or not) the expected results as per Log-frame?

Regulatory framework:

1. MoH order (#185) on

Regionalization, 2014

o What are the challenges in

implementation of the

perinatal referral system

(probe for: service level

requirements, criteria

adjustment to country

context, costs by perinatal

service level, financing,

equipment, HR sufficiency, HR

skills, transportation system,

communication)

o How the facilities were

assigned level of care (probe

for geographical fact,

accreditation, licensing)?

o What steps were undertaken

/ are planned to address the

challenges?

o Are any tools developed to

monitor the referral system?

2. Update of MoH order#183 on

Organization of Neonatal Care

(draft in May, 2015)

o Was it approved?

o Does it required update on

referral criteria (link to the

order #185)

3. Clinical guideline for inpatient

care for children under age of 5.

MoH Order No. 225 10 July 2013

4. Clinical guideline on outpatient

care for children under age of 5

years.

o Was it approved?

5. Updated training curricula of

medical Universities. MoE Order

No. 319 from 13.08.2013.

6. Neonatal Resuscitation

guideline. MoH order

03.02.2015

X X X X X

43

7. National standards on obstetric

care in maternities revised by

WHO expert.

o What is the status?

8. National standard on NMCR and

CEMD. July 2015

9. Revision of MoH Order No 28 of

25 January, 1993 on

Methodology of Development

and Analysis of Chid Mortality,

o what is the status of draft

order?

10. National policy and tools on

supportive supervision,

o what is the status?

Pre and Post graduate education

o Training courses are fully

integrated at undergraduate

level, however at post graduate

level the integration is

fragmented, what are reasons for

that? How this could be

addressed?

o Are there obligatory list of

courses for specialties, e.g. for

pediatricians to accumulate

required number of courses?

o Does Curricula for midwifes and

nurses (undergraduate level)

include full MCH package?

X X X X X

MCHSCC and secretariat

o How effective is functioning of

the MCHSCC (frequency of

meeting)

o Who finances Secretariat?

o How effective is secretariat?

o How this function will be

sustained in the future?

X X X X X

National Supportive Supervision (SS) System?

o What is the WHO role in

development of Supportive

Supervision?

o Which staff participate? (probe

on MoH, trainers, regional health

dept.)

o What is coverage of supportive

X X X X

44

supervision?

o Are standard methodologies

used for local supportive

supervision plans?

o How feedback from SS is given?

o How changes are monitored?

o Could you describe examples

when supportive supervision

results were not used for

punitive purposes?

Fulfilment JMT recommendations

(II period): Equipment

o Is there an Inventory of Basic

Neonatal Care equipment

o Are service books for each item

of equipment Implemented;

o Centralisation of procurement of

equipment, spare parts and

consumables at national level.

o Is Equipment maintenance policy

developed (role of KfW)

X X X X X X

National Concept on Quality Improvement.

o Is the Concept and Action plan

finalized? Consensus reached?

o Are Tool for certification of

health facilities, developed? If

not what are reasons of delay?

Please describe role of GIZ

o Please describe Four step

approach for Quality

Improvement? What is role of

KFW, WB-3 project, GIZ?

o WHO Pilot the Health System

Strengthening activities in two

pilot regions, what does this

activities entail?

X X X X X

HIS

o What are the benefits of the

child birth and child mortality

databases

o Please describe for what

purposes do you use the data

(provide examples)

o Have you planned any activities

base on data analyses

o Are instructions available for

database application

X X X X

45

o Are there any challenges with

the database?

Training database.

o Does MoH use this information

for planning of other (not

project supported) training

activities, if yes please list them

and what is funding source?

o How user-friendly is the

database, any shortcomings?

o Are there instructions available

for database application?

o From which sources the

operation of the database is

supported?

o From which sources the

operation of the database will

be supported after completion

of the project?

X X X

Training centres

o Training materials are

standardised and MoH

approved. Do partners use the

same training materials for their

training courses?

o Problems with selection of

training participants, how it was

addressed?

X X X

Community Behavioural Change mechanisms using PLA approach:

o What are criteria for assessing

successful functioning of PLA

(defined by the BCC guideline)?

o Have behaviour change

guidelines been adopted?

o What are challenges of PLA

approach (probe for design

appropriateness, considering

local context, patronage nurses

workload)

o What were reasons for PLA

delayed implementation?

X X X X

Q5 Could you bring examples of policy change as a result of lessons learned from the project implementation? (if not covered by question Q4):

X X X X

Q6 There are improvement in provider’s X

46

knowledge and practices as shown by baseline and endline Health Facility Surveys. Please describe

o what changes do you observe in

your facilities

o what should be done to sustain the

changes

o what are challenges for further

improving of providers knowledge

and practices

Q07 o Could you describe any changes or

corrections done during project

implementation based on JMT or

supportive supervision results?

o Could you describe any managerial

actions done based on reviewing

facility/rayon child birth/mortality

data

X X X X X

Q09 Where there are any unintended positive and/or negative results and whether the negative results could have been foreseen and managed? Probe on leveraging of resources from oblast health departments and Perinatal centers to allocated funds for sustaining the training and Supportive Supervision.

X X X X X

Q10 Could the intended results been achieved at a higher level of quantity / quality by:

o Better responsiveness and

flexibility of the Project

management

o Improved monitoring of risks and

external factors

o Targeting of health professionals

during trainings

o Shifting balance of

responsibilities between the

various stakeholders

o Accompanying measures taken

or to be taken by the

government

X X X X X

Q11 o Were the available resources

adequate to meet project

objectives? Probe for:

o All planned activities

implemented within available

X X X X X

47

budget

o All planned activities

implemented were adequately

financed

Q12 Could you describe by each project activity (trainings, PLA, databases ) which Governmental institution has ownership on its implementation?

X X X

Q13 o Will the government continue

implementation of project activities

with own resource or find external

funds for: MCHCC, training activities,

supportive supervision, PLA

approach?

o Is required budget estimated?

o Are funds allocated?

o What is MCH budget for 2016

(increase, stable)

X X X X X

Q14 To what extent is the project facilitated synergies and avoided duplications with interventions and strategies promoted by other UN agencies and developing partners?

X X X X

Q15-Q16

o Please describe whether HR & GE are

clearly reflected in the intervention

design (log frame, indicators,

activities)

o Please describe whether

intervention design benefitted from

specific human rights and gender

analyses

o Please describe whether the JM

forms and reporting forms include

gender, vulnerable groups

X X X X X

48

ANNEX 6: FGD GUIDES

ANNEX 6.1: FGD GUIDES FOR SERVICE PROVIDERS (Obstetricians, Pediatricians, Neonatologists, Nurses, Midwives, Patronage Nurses, GPs)

1. Introduce yourself 2. Introduction to the objectives of the research 3. A brief introduction to the rules of focus groups

a. Everything said and done is confidential and will not be used outside the room except for the purposes of this research;

b. Every statement is right; c. Please do not hesitate to disagree with someone else; d. But do not all talk at once

4. Ask people to describe who they are and say few words about themselves 5. Introduce the topic under review - We are here to evaluate the training and monitoring

component supported by the IMCHS Project 6. Ask questions

Relevance

In your opinion how the new technologies introduced and provided through the training

addressed your needs?

Why do you think that training you received is pertinent to your current daily work?

Please explain whether the new way of service provision meets women, children and

their families?

Effectiveness

In your opinion how the service quality has changed in your facility after trainings and

what are arguments supporting your judgment?

What is the difference in the way you provide services today with the way services were

provided before the training?

What helped and/or impeded you to accept new practices/procedures (reluctant to

change, lack of required medical equipment, job aids, etc.) and explain how?

Please explain whether and how supportive supervision helps/limits you to improve

service quality?

What data do you usually use for decision-making? Please give an example

Please explain what is your involvement in the monitoring process?

Impact

IMCHS project focuses on improving the quality of Mother and Child health care. In

doing so, what are improvements the project brought?

How the project facilitated improved management of health facility and resource

mobilization? Please give examples

Sustainability

Which factors prevent you to correctly practice your skills? (I.e. non-confident in skills

despite training, shortage/lack of basic equipment/amenities, drugs, time constraints,

referral etc.). Please, describe.

Do you expect to be incentivized/awarded for delivering quality MCH services? If not,

how it affects the service provision?

49

7. Ask if they would like to add further comments.

8. Bring the meeting to a close by summarizing the main points.

9. Thank you

50

ANNEX 6.2 FGD GUIDE FOR BENEFICIARIES

1. Introduce yourself 2. Introduction to the objectives of the research 3. A brief introduction to the rules of focus groups

a. Everything said and done is confidential and will not be used outside the room except for the purposes of this research;

b. Every statement is right; c. Please do not hesitate to disagree with someone else; d. But do not all talk at once

4. Ask people to describe who they are and say few words about themselves 5. Introduce the topic under review - We are here to evaluate the training and monitoring

component supported by the IMCHS Project 6. Ask questions

Relevance

What kind of services and information do you want or need to receive for yourself and/or

your child?

Is the communication package relevant to your demands?

Effectiveness

What is the primary form of communication that moves you to action? (Face to face

communication in the health facility and/or at home, Flip chart on key child feeding, caring

and health seeking practices; Mother card; Posters; Examples; other)

Please give examples when you apply acquired skills and knowledge into practice.

How would you assess your last visit to MCH service provider in terms of:

o Content of counselling and treatment (where applicable);

o Responding to all your questions;

o Provision of sufficient information about the problem, diagnostics and treatment?

Sustainability

What are the main problems you face in getting the quality MCH services for mothers and children in a timely manner (socio-cultural, accessibility (physical and financial), acceptability, other)?

51

ANNEX 7: TERMS OF REFERENCE

Terms of References, UNICEF Uzbekistan

Summative Evaluation of Improvement of Mother and Child Health Services (Phase 2) in Uzbekistan (July 2012 – December, 2015)

I Background and Context

Uzbekistan became a lower-middle income country in 2010 with a child population over 10 million. Despite reported decrease in national income poverty rates from 26.1 percent in 2004 to 15 percent in 2012, there are still some disparities particularly in rural areas and in Karakalpakstan, Kashkadarya and Surkhandarya regions51.

It has been estimated, that in 2012 Uzbekistan spent 5.9% of its gross domestic product (GDP) on health52. Public expenditures accounted for 53% of health expenditures and remaining 47 % are funded from the out-of-pocket expenditures. The government provides guaranteed basic benefit package that covers primary care, emergency care and care for ‘socially significant and hazardous’ conditions, including major communicable diseases and some non-communicable conditions53.

In 2013, the UN Inter-agency Child Mortality Report estimated under-5 mortality at 43, with infant mortality at 37 and neonatal mortality at 14 per 1,000 live births54. The official data from State Committee on Statistics (SCS) reports under-5 mortality at 13.4 per 1000 live births, with infant mortality at 9.8. Infant mortality rates are higher in rural areas and among children from the poorest quintile. The most prominent disparities are found between regions55. Likewise, the UN Maternal Mortality Rate Estimation Interagency Group reported MMR at 36 per 100,000 live births in 2013, while official sources reported MMR at 20 per 100,000 live births. There are a number of factors that contribute to discrepancies between the national and international estimates, including that Uzbekistan does not use the WHO recommended international life-birth definition (ILBD) 56. In January 2014 the Ministry of Health (MOH) issued a decree (#21) on adoption of the international life-birth definition (ILBD) in Uzbekistan. However, the ILBD yet to be applied to the data reported by the national statistics. High maternal and neonatal mortality rates, despite high coverage of antenatal care and skilled attendants at birth, suggest major issues with quality of healthcare services. According to 2013 official data, child mortality has reduced from 12.5 per 1,000 live births in 2008 to 9.8 in 2013. However, neonatal mortality did not change significantly. Likewise, according to 2010 WHO data, around 49 per cent of maternal deaths are directly related to obstetric factors or incorrect management of complications57. The UN Committee on the Rights of the Child that the quality of maternal, perinatal and early neonatal care in Uzbekistan is inadequate58. The results of baseline assessment of maternal and child care services in 2013, showed that the healthcare system needs to be strengthened to provide adequate quality of continuum of care.59. In addition, the concept of quality monitoring is relatively new to the system; there is no clear set of quality indicators, milestones and targets on quality of healthcare services. Lack of proper knowledge on good child-rearing practices to support children’s survival, early development in the families and especially the capacity of caregivers to recognize danger signs of childhood disease is another factor contributing to infant and child mortality. There is little recent data

51 Uzbekistan Millennium Development Goal Report 2015 52 The average of government spending as percentage of GDP in WHO European region was 8.3 percent. 53 Health Systems in Transition, Uzbekistan, WHO 2014 54 UN Inter-agency Group for Child Mortality Estimation, Level & Trends in Child Mortality, 2013, 55 GoU and UNICEF Uzbekistan, Multiple Indicator Cluster Survey (MICS), 2006, 56 European Observatory on Health Systems and Policies. Uzbekistan: Health System Review, Vol. 9, No. 3. 2007. 57 WHO regional estimates for CIS (1997-2007), 2010 58 United Nations, CRC/C/UZB/CO 3-4 Concluding observations on the combined 3rd&4th periodic reports of Uzbekistan, 2013 59 United Nations, CRC/C/UZB/CO 3-4 Concluding observations on the combined third and fourth periodic reports of Uzbekistan, adopted by the Committee at its sixty-third session (27 May-14 June 2013), available at http://www2.ohchr.org/english/bodies/crc/docs/co/CRC-C-UZB-CO-3-4.pdf

52

available but, for example, in 2006 only 15 per cent of caregivers were able to recognize two symptoms of pneumonia.60 In addition, there are some public health issues affecting child survival, growth and development. UNICEF global estimates for 2008-2013 show stunting at 20 per cent and underweight at 4 per cent among young children. Stunting and underweight is more prevalent among children from poorest quintile. 61 According to 2012 nationwide household survey, anemia was a moderate health problem using WHO criteria, with 34.4 per cent prevalence of anemia among women of reproductive age.62 Over the past decade several major health reforms have been implemented in Uzbekistan under the framework of State Program of Reforming the Health Care System aiming to improve efficiency of health care system and ensure equitable access63. The major focus of these reforms has been on health care provision, improving the infrastructure and hardware component, building the competency of health care providers, and rationalized financing and governance64. The Presidential decree issued in November 2011 “On Measures to further Health Care System Reforms” identified quality of health care services and improvement of maternal and child health (MCH) services amongst the key priorities for 2012 – 2015. Besides, the State Programme (SP) on Strengthening and Development of System of Protection of Reproductive Health of Population, Health of Women, Children, and Adolescents in Uzbekistan for years 2014 – 2018 aims to improve the following major areas: (i) Equity and accessibility of quality medical services; quality of medical and social rehabilitation of children with disabilities and their social inclusion; (ii) Potential of human resources at all levels of the health care system, especially at the level of primary health care; (iii) Infrastructure of health care institutions and available technologies; (iv) Active participation of population in care for mothers and children, adoption of health behaviors, and promoting healthy families; (v) Informational health management systems, coordination and inter-sectoral cooperation. In order to improve maternal and child health and support Uzbekistan to meet Millennium Development Goals (MDGs 4 & 5), "Improvement of Mother and Child Health Services (IMCHS)" phase I project was designed and implemented in eight regions of Uzbekistan (Samarkand, Sirdarya, Namangan, Djizzak, Navoi, Surkhandarya, Kashkadarya, and Andijan) in 2008-2011, aiming at improving health care providers' skills on quality of care. Project's Specific Objective was to support the implementation of Uzbekistan's national healthcare reforms through strengthening newborn care and improving the quality of maternal and child health care by developing skills and capacity in prenatal and newborn care at the hospital level and the management of childhood diseases at the primary health clinics The expected results of the phase I were: 1) Improved skills of staff on effective perinatal care, newborn and child care in maternities, village level health facilities (SVP) and polyclinics; 2) Improved quality of care during delivery and post natal care in all hospitals and maternity centres; 3) WHO "live birth definition" universally applied; 4) Reporting and monitoring system of births improved; 5) Pre-service curriculum updated and brought in line with training programme (including teacher training), introduction (or piloting is initiated); 6)Public awareness of "best practice in child care in general" is raised. Phase I “Improvement of Mother and Child Health Services (IMCHS)” project brought scalable and cost-effective solutions to providing equitable access to health services for mothers and children. But more importantly, it gave compelling evidence that an integrated package of newborn and child survival interventions along the continuum of care from pre-pregnancy to childhood can significantly reduce child morbidity and mortality, thus bringing the country closer to attaining Millennium Development Goals 4 and 5. With a view to sustain and expand the project phase I achievements, the Ministry of Health decided to take this innovative approaches to a nation-wide scale-up through a second phase of the project to boost the nationwide expansion and focus on institutional strengthening, capacity development and community empowerment for equitable, quality and continuous health care. In July 2012 the Agreement was signed between Ministry of Health, UNICEF and the European Commission to mark the roll out of the IMCHS

60 UNICEF Uzbekistan and Government of Uzbekistan, Multiple Indicator Cluster Survey (MICS), 2006 , at http://www.childinfo.org/files/MICS3_Uzbekistan_FinalReport_2006_Eng.pdf 61 United Nations, Common Country Assessment: Uzbekistan, 2014 62 GAIN, Report to Assess the Result of National Flour Fortification Programme: LC-LQAS Survey Report, 2013 63 Health Systems in Transition, Uzbekistan, WHO 2014 64 Ibid 61

53

Phase 2 with a duration of forty two months and it is implemented in the remaining six regions to complete nationwide scale-up: Fergana, Tashkent, Bukhara, and Khorezm Oblasts, the Republic of Karakalpakstan, and Tashkent city. II The object to be evaluated The evaluation will assess the entire “Improvement of Mother and Child Health Services (IMCHS)” Project, Phase 2 of the project as described below and detailed in the logical framework (Annex 1) and the Inception Report summary (Annex 2) Project Synopsis: Project title Improvement of Mother and Child Health Services in Uzbekistan –

Phase II

Implementation period July 25 2012 – December 31, 2015

Total contracted amount EUR 4.900.000

UNICEF Contribution EUR 818.770 (in addition to the EU funding)

Target country and regions The Republic of Uzbekistan (Fergana, Tashkent, Bukhara, and Chores Oblasts, the Republic of Karakalpakstan, and Tashkent city)

Project partners Government of Uzbekistan

Ministry of Health

WHO Country Office in Uzbekistan

Target groups At national level: Ministry of Health and Republican Medical Institutes

At regional level:

(i) 4 Oblast Health Departments (Fergana, Bukhara, Khorezm and Tashkent), Ministry of Health Republic of Karakalpakstan and Health Department of Tashkent city

(ii) 13269 health care professionals (771 Trainers,560 Managers, 11938 Physicians and Nurses)

i) (iii) 12 Rural communities in pilot Oblasts Final beneficiaries 15.5 million population of Fergana, Bukhara, Khorezm and

Tashkent Oblasts, Republic of Karakalpakstan and Tashkent city, including 3.5 million children under the age of 5 years

The overall objective of the Project is to contribute to the human development in Uzbekistan with special emphasis on Mother and Child Health Care (MDGs 4 and 5). The specific objective is to “support the MoH to implement MCH care sector reforms, increase the quality of MCH care services, and develop the capacity of families to adopt healthy behaviors”. While more specific information on the intervention logic is provided in the inception report and in the logical framework, the below graph attempts to show the logframe in a concise manner for easy reference as well as to link it with the theory of change logic (using the RBM chain of activities – outputs – outcomes – impact) used in UNICEF’s programming.

54

The phase two project intent to achieve 2 main results: Result 1. Institutional strengthening. Skills and operational capacity of the Ministry of Health and its three tiers (primary, secondary, and tertiary) have been enhanced to effectively support the health reform process concerning the Mother and Child health services in accordance with international standards. Activities: 1.1. Support the MoH to establish and operate the Maternal Child Health Coordination Council (MCHCC), its secretariat, and Joint Monitoring Team (JMT). 1.2. Establish a formal certification system and process for the MCH care institutions. 1.3. Elaborate and pilot quality improvement mechanisms for the MCH care institutions 1.4. Pilot the Health System Strengthening activities in two pilot regions. 1.5. Perform need assessment, elaborate specification of essential medical equipment for provision of quality EPC, NR&ENBC services. Result 2. Capacity development and empowerment. The capacity of health workers to provide quality health services is improved. The capacity of families to adopt healthy behaviors and demand better health services is enhanced Activities: 2.1.1. Conduct training and supervision of healthcare providers on maternal, newborn and child survival packages as per the approved standard protocols and guidelines. 2.1.2. Support the MoH to implement the newborn survival, child survival, and nutrition modules in graduate and postgraduate curricula for MCH care professionals. 2.1.3. Train health care system managers to support improvement of MCH care services. 2.1.4. Establish a platform for discussion and exchange of experience and dissemination of the best practices on MCH care sector reform issues. 2.2.1. Establish community based behavior change mechanism based on participatory learning action approach (PLA) under the ownership of selected Mahallas and primary healthcare institutions (SVP). 2.2.2. Develop PLA modules, BCC materials and tools which promote healthy behaviors, including nutrition, national awareness, and visibility activities. 2.2.3. Train patronage nurses and Mahalla Advisors from Women’s Committees to act as facilitators in implementation of community activities.

55

2.2.4. Create support groups for promoting healthy behaviors and monitoring of outcomes of community based BCC activities. Project Partners: The main international development partners, their role in healthcare system reform in Uzbekistan, their respective contributions, and collaboration with the project described below: 1. The World Health Organization (WHO): The World Health Organization (WHO) Country Office

provides technical support to the IMCHS Project Phase II at national level, and in implementation of the health system strengthening of provincial and district health departments in two pilot regions (Namangan and Republic of Karakalpakstan).

2. The Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ): The project has collaborated with GIZ through joint activities on quality improvement, piloting of certification of MCH care institutions and BCC community activities in Republic of Karakalpakstan (RoK).

3. The Kreditanstalt für Wiederaufbau (KfW): KfW has cooperated with the project in implementation

of the JMT visits. The project has agreed with KfW/GFA to cover the H-IMCI training activities. KfW/GFA will be responsible for the implementation of training activities in specialized pediatric care, and for the development of related clinical protocols in pilot Oblasts. The project provided trainings on use of Neonatal Equipment (NET) to 14 Children Oblast Hospitals equipped by KfW.

4. The World Bank (WB): The project collaborated with the WB Health 3 project through joint

implementation of IMCI training activities for doctors and nurses at the Oblast level. It was agreed that the WB Health 3 Project will implement the H-IMCI training activities in the pilot Oblasts of the IMCHS Phase I Project, and will train the nurses from the Oblast Children’s Multi-profile Medical Centres in all Oblasts of Uzbekistan.

5. The United Nations Family Planning Association (UNFPA): has cooperated with the project in

implementation of JMT and Near Missed Cases Review (NMCR) approach. The project and UNFPA have agree to implement their activities in different regions, to prevent duplication and overlapping. The project agreed with UNFPA to cover the Effective Perinatal Care (EPC) training activities. UNFPA will be responsible for the implementation of Emergency Obstetric Care (EMOC) training activities. UNFPA has supported an assessment of national perinatal centres in terms of readiness for EMOC by an international consultant. It was agreed that the report of the international consultant had been incorporated into the BAQMCHS as undertaken by the project. In addition, the project facilitated the linkages b/w the UNFPA and LDS Charities to implement the Helping Mothers Survive training (component related to obstetric hemorrhages).

6. The United Nations Children's Fund (UNICEF): is seen by Government as the subject-matter

expert on mother and child health services and as the convener of partners that support the MCH sector reform. UNICEF is working with the MoH of Uzbekistan to improve access to and quality of health care for mothers and children by working at different levels: policy, system and community. UNICEF is an implementing entity of the IMCHS phase II project in close partnership with MoH and EU Delegation in Uzbekistan.

In December 2012 MOH established (Order No. 359 from December 18 2012) the Maternal Child Health Coordination Council (MCHCC) to coordinates the efforts of the MOH, and relevant national health institutions (Republican Specialized Scientific and Practical Medical Center of Pediatrics, Republican Perinatal Center, Institute of Health and Medical Statistics) and international partners and donors (EU, UNICEF, UNFPA, WHO, KfW, GIZ, WB) in the area of MCH. During the implementation of the ‘Improvement of Mother and Child Health Services in Uzbekistan – II Phase’ project, this mechanism also serves as a project Steering Committee, to help position the project within the strategic framework of health sector reform. A secretariat has been established to support the MCHCC’s effective functioning, and two technical working groups are in charge of programmatic and medical education issues

56

III Rationale for the evaluation This summative evaluation will come at a time when: (a) The IMCHS project, phase II will conclude its activities, and (b) The Country Programme of Cooperation between the Government of Uzbekistan and UNICEF will be

finalizing the 2010-2015 cycle and will embark onto implementation of a new 2016 -2020 Country Programme.

In this context, the Evaluation offers the opportunity to critically assess IMCHS Phase 2 as a stand-alone project as well as in the broader context of scale up of cost-effective interventions related to the maternal, newborn and child health care and their contribution to maternal and child health and well-being. The findings of the evaluation will be used as a basis for discussions, planning and programming between UNICEF and national actors, as well as by international entities, in particular:

1. The Government of Uzbekistan and UNICEF to inform the strategies to be applied in the new Country Programme 2016-2020 will use the knowledge generated by the evaluation.

2. The knowledge generated by the evaluation will be used by key relevant stakeholders and international development agencies (MOH, UN Agencies, etc.) represented in the Mother and Child Health Coordination Council (MCHCC) to inform policy making and to further support MCH reform agenda for improving quality of MCH services.

IV Objectives of the Evaluation The main objectives of this summative evaluation are to:

Assess the relevance, efficiency, effectiveness, sustainability, coherence and, to the extent possible, impact of the project results;

Identify and document successes, challenges and lessons learnt;

Provide recommendations to guide: (a) implementation of the next program cycle, and (b) policy level decision making by relevant stakeholders and international development agencies

V Scope of the Evaluation As mentioned, the evaluation will assess an entire project and will cover most of the Phase 2 implementation period, from July 2012 to December 2015. The geographical scope will include the project target regions (as indicated above), and both national and sub-national levels. Furthermore, the evaluation will focus on the criteria of relevance, effectiveness, efficiency and sustainability and coherence. The impact will also be assessed, to the extent possible. One of the limitations that might hinder the evaluation process in some areas is the limited availability of reliable data related to the mother and health services, especially its disaggregation across different vulnerable groups, which may limit the assessment of equity dimensions. Although all efforts have been made to systematically document design and implementation of the project, some project interventions might require collecting additional information that also might hinder the evaluation. VI Evaluation Questions/Framework In general, the evaluation should aim at answering the below questions. However, further details will be discussed during the Inception Phase and questions may be fine-tuned based on considerations of evaluability. Assessing relevance

To what extent were the project design, strategy and approach appropriate to achieve the set objectives?

To what extend was the project implemented in partnership with the relevant stakeholders? And at the right level (local, national)?

To what extent was the project relevant in terms of contributing to improve the health and well-being of mothers and children?

57

To what extent was the project inserted into a broader context and designed to contribute to the MCH sector reform?

Assessing effectiveness

To what extent has the project contributed to achieving (or not) the expected results as per log-frame?

To what extent the project interventions have been effective in facilitating MCH sector reforms with respect to policy change and quality improvement?

How effective the project interventions have been in improving service providers’ knowledge and practices in New-born and Child Survival packages

In the MCH facilities where trained service providers work, to what extent regular practices have been modified with relation to improvement of quality of care? What are the enabling/constraining factors that facilitated/hindered this change?

To what extent is the M&E system (including supportive supervision) has been effective in reinforcing skills application and tracking progress?

To what extent have beneficiaries from pilot regions improved their child care and health seeking practices as a consequence of community based activities?

Assessing efficiency

Has the IMCHS Phase II project used the resources in the most economical manner to achieve its objectives?

Were the available resources adequate to meet project objectives?

Assessing sustainability

To what extent do the MOH and other concerned health institutions demonstrate ownership over different project components?

To what extent the MOH takes the ownership over the New-born and Child Survival packages to ensure sustainability of the achieved results through policy, regulatory framework and capacity development?

To what extent the communities demonstrate the ownership and capacity to sustain behavior change component?

Assessing impact

To what extent have the project activities contributed to achieving (or not) the expected impact level results (mortality, etc.)?

Assessing coherence

To what extent is the project facilitating synergies and avoiding duplications with interventions and strategies promoted by other UN agencies and development partners within the MCH sector and its reform?

In addition to the main evaluation criteria, the evaluation shall also focus on assessing human rights-based approach (HRBA) and relevant crosscutting issues:

To what extent and how did the project incorporate the HRBA?

To what extent and how did the project incorporate gender equality? VII Methodology of the Evaluation

58

The Norms and Standards of the United Nations Evaluation Group (UNEG) will guide the evaluation methodology65

Evaluability Assessment

This is a preliminary evaluability assessment. At inception stage, the evaluator(s) are expected to conduct

a thorough review and analysis of secondary data available in order to identify information gaps and other

evaluability challenges and discuss solutions to address these.

In general, the various reports and available data allow for the assessment from the point of view of the different criteria, thought assessing contribution to impact may present some challenges especially in terms of impact on specific vulnerable groups. The documents listed below provide background information, baseline, mid-review and end-line quantitative data as well as qualitative information. Reliability of data, especially disaggregated, is an issue to be taken into account. The UNICEF Country Office will be able to provide more specific guidance on this issue during the inception phase. While data gaps are not the general rule, in some cases they may hinder evaluability. In these cases, during the inception phase, the evaluation team is expected to agree with the commissioning team on alternative approaches, including the use of less rigorous evaluation designs and/or the selection of the evaluation questions that can indeed be answered.

Information sources

The following list includes general information sources related to country context, health sector and the project: Background/situation monitoring sources: - Uzbekistan Common Country Assessment (UN, 2014)

- Statistical Year Books 2012, 2013 year - Health in Transition, WHO Observatory, 2014 - MICS Uzbekistan 2006 - Country Programme Document 2010-2015 (Government of Uzbekistan, UNICEF, 2009) - Country Programme Action Plan 2010-2015 (Government of Uzbekistan, UNICEF, 2009) - UNDAF 2010-2015 (Government of Uzbekistan, UN, 2009) - Country Programme Mid Term Review (MTR) Report (Government of Uzbekistan, UNICEF, 2013) - Country Office Annual Reports 2010, 2011, 2012, 2013 - MCH State Program (Country Strategy) 2009-2013, 2014 – 2018 - MOH Decrees and normative documents related to MCH services (Annex 3)

Planning and project monitoring sources:

- IMCHS Inception Report - Project progress reports 1, 2 and 3 - Midterm Review - Baseline and End-line Health Facility Assessment (BLA) - Mid-term formative evaluation report - KAP Baseline and Final Report - PLA modules

Evaluation Approach

The evaluation will be conducted in a participatory manner and participation of key stakeholders will be ensured in all phases of the evaluation, including the planning, inception, fact-finding, reporting as well as the management response phases. To this extent, an Evaluation management team comprised of representatives of Ministry of Health, EU, WHO, UNICEF Health Section Programme Staff and M&E officer will lead the evaluation throughout the entire process. The Evaluation management team will

65 UNEG Norms http://www.uneval.org/document/detail/21

UNEG Standards http://www.uneval.org/document/detail/22

59

coordinate overall the involvement of all stakeholders at key milestones through MCHCC (MOH, WHO, UNFPA, KfW, GIZ, WB, and others), to allow general consensus and guarantee ownership over the findings. The overall evaluation approach will be based on the theory of change spelled out in the logical framework and detailed in the project inception report. Depending on the project component to be measured, the evaluation will have to use different approaches. For Result 1, non-experimental designs are to be preferred. For the component related to health providers in Result 2, a quasi-experimental design should be possible. The community-based component in Result 2, will most probably require a non-experimental design.

Data collection will be based on a multiple method approach, including:

1. Desk review of:

Planning documents

Monitoring documents, including: a. baseline, mid-term and final health facility assessment; b. baseline and final KAP study on household behaviours c. other monitoring documents and data base (including internal, joint and external

monitoring activities);

Accessible policy and planning documents on the maternal and child health sector as well as relevant documents on sector’s assessments and capacity gap analysis.

Available administrative and survey data on relevant indicators.

2. A mix of In-depth interviews and Focus Group Discussions (while the FGDs may not be the most effective approach due to country-specific social norms, it may still be discussed case by case) to collect qualitative information within key health care providers, health managers, mahalla (community) leaders, caregivers, national course directors, Ministry of Health’s officials, regional governments and regional departments of health, donors and development partners

Triangulation of data (combining qualitative and quantitative data as well as data from a range of stakeholders) will have to be used to increase reliability of findings and conclusions. Adequate measures will be taken to ensure that the process responds to quality and ethical requirements. Interviewees should be protected (e.g. references to information sources should remain confidential and the report will not contain names unless explicit permission is granted). During the Inception Phase, in consultation with the Evaluation management team, the sample of stakeholders to be interviewed and locations to be visited will need to be defined based on agreed criteria. At this point, the evaluation questions will be refined; the evaluator(s) should also develop a more precise evaluation work plan. VIII Work plan of the Evaluation

The evaluation process will consist of three phases:

1. Inception phase including:

In-depth desk review of available sources so that the evaluator(s) improve their understanding of related programme areas, involved stakeholders, and the country context

Preliminary discussions with the Evaluation management team and other relevant actors, to facilitate an in-depth common understanding of the conceptual framework;

More in-depth evaluability assessment

Refining the evaluation questions and adjusting data collection methods and sample;

Inception report preparation, including: Evaluation Matrix for each finally agreed evaluation question, data collection and analysis methods, sample (list of stakeholders to be interviewed and locations to be visited), and operational plan. The inception report will have to be shared with and approved by Evaluation management team based on the criteria set by UNICEF evaluation quality assurance system.

2. Data collection phase, including an appropriate mix of data collection methods, as indicated above.

This phase will have to be partially conducted in-country. Analysis and reporting phase. Following the completion of the fact-finding and analysis phase, a draft report (in English) should be shared with Evaluation management team and submitted for discussion to the Maternal and Child Health Coordination Council (MCHCC). Following the review and comments

60

received, the draft report will have to be finalized and approved by Evaluation management team based on the criteria set by UNICEF evaluation quality assurance system. The process will be guided by the following schedule (expected to take place in September-November 2015)

Activity Timeframe Location Deliverables

Inception Phase: 13 working days

Preliminary desk review and discussions with UNICEF team

5 working days Home-based Inception report

Refining evaluation questions and in-depth evaluability assessment

3 working days Home-based

Preparation of the inception report 5 working days Home-based

Data Collection Phase: 19 working days

In-depth desk review to gather secondary quantitative/qualitative data

5 working days Home-based Data collection and analysis tools

Preparation of data collection and analysis tools

5 working days Home-based

In-country data collection 10 working days Uzbekistan Presentation of preliminary findings

Analysis and Reporting Phase: 20 working days

Data processing and analysis based on all information collected

10 working days Home-based Database of data collected

Preparation of draft evaluation report 5 working days Home-based Draft report

Consolidating UNICEF comments and preparation of the final report

5 working days Home-based Final report

IX Evaluation Management

The Evaluation management team will oversee the management of the evaluation process starting from the development and validation of the present terms of reference, selection of the evaluator(s), liaison between the evaluator(s) and partners / stakeholders involved, ensure quality of the report and determine the management's response to the evaluation findings and recommendations. It will coordinate with key stakeholders through MCHCC at key milestones such as inception and data collection stage, report validation and discussion of findings and recommendations. It will also ensure operational support as required, including support in primary data collection where needed to complement what available from the existing monitoring systems and other documents. Required qualifications and areas of expertise The evaluation will have to be conducted by a gender-balanced team comprising a sufficient number of qualifying international evaluators covering the below requirements:

Team-leader with documented extensive experience (at least 8 full years) in conducting complex development evaluations (having conducted evaluations for UNICEF is an asset, having evaluations positively rated by UNICEF’s quality assurance system is an additional asset);

Other evaluator(s) with documented experience (at least 5 full years) in conducting development evaluations (having conducted evaluations for UNICEF is an asset);

At least one team member with proven extensive experience in quantitative and qualitative data collection and analysis;

All team members with experience of working in developing countries, at least one team member with experience in Commonwealth of Independent States (CIS) (previous work in Uzbekistan is an asset);

At least one team member with solid knowledge on mother and child health;

At least one team member with solid knowledge of child rights, HRBA and gender equality

Excellent report writing skills in English;

Good communication skills

Fluency in English, fluency in Russian is an asset.

The team should be an international institution and it may be complemented by one or more national consultants for support in translation, organization of the in-country agenda, and interpretation of findings from a country-specific stand point if needed. To this extent, it should be kept in mind that there is no

61

evaluation society in Uzbekistan and it would be more realistic to expect the national team member(s) to be consultants rather than evaluators.

The team is responsible to ensure that the process is in line with the United Nations Evaluation Group (UNEG) Ethical Guidelines http://www.uneval.org/document/detail/102. The evaluator(s) should be sensitive to beliefs, manners and customs and act with integrity and honesty in their relationships with all stakeholders. Furthermore, they should protect the anonymity and confidentiality of individual information. All participants should be informed of the context and purpose of the evaluation, as well as of the confidentiality of the information shared.

The evaluator(s) are allowed to use documents and information provided only for the tasks related to these terms of reference.

X Deliverables, including Structure of the evaluation report As described in the last column of the matrix in “Work Plan of the evaluation”, the expected deliverables are the following:

Inception report – to be delivered 13 working days from the start of the contract;

Data collection and analysis tools - to be delivered 23 working days from the start of the contract;

Presentation of the preliminary findings - to be delivered at the end of the in-country mission, 33

working days from the start of the contract;

Draft report - to be delivered 48 working days from the start of the contract;

Final report - to be delivered 53 working days from the start of the contract.

The Evaluation Report should comply with UNICEF Evaluation Report Standards as outlined at http://www.unicef.org/evaldatabase/files/UNICEF_Eval_Report_Standards.pdf. The report should include:

Executive summary,

Description of the object of the evaluation (including theory of change and relevant information),

Purpose of the evaluation, evaluation scope, objectives and criteria

Description of the evaluation methodology (including evaluability assessment, limitations and ethical issues),

Findings broken down by evaluation criteria,

Conclusions and lessons learned,

Recommendations,

Annexes, including: Terms of Reference, data collection tools and other relevant information. The quality of final evaluation report will be assessed by external independent company in the framework of UNICEF Global Evaluation Reports Oversight System (GEROS). XI Procedures and logistics

The evaluators will be assisted with logistics related to the assignment. During in-country visits, they will be provided with office space, vehicle for site visits and official meetings, logistic support for meetings and VISA procedures. Laptops or computers will not be provided. XII Payment schedule

Payments shall be made as follows:

30% will be paid upon submission of the Inception Report;

40% will be paid upon presentation of the preliminary findings;

30% will be paid upon submission of the Final report; Please note: in compliance with national Uzbek laws, no cash will be paid in the country

XIII Resource requirements

The rate per day of professional fees will be in accordance with the complexity of the TOR and the level of the expertise required, which is estimated at P4 and P3 level.

Tentative budget for internal planning purposes estimated based on a team of 3 evaluators (1 team leader at P4 level, 2 team members at P3 level):

62

Cost category Cost description Total, USD

Fee Team Leader – P4 $ 500 x 50 days $ 25,000

Fee Team Member 1 – P3 $ 400 x 40 days $ 16,400

Fee Team Member 2 – P3 $ 400 x 35 days $ 14,000

Travel (ticket) Round trip ticket cost (by air) $2000 x 3 $ 6,000

Travel (DSA) $190 x 30 nights (24+trip) $ 5,700

Terminal expenses $120x3 $ 360

National consultant $ 1000 x 2 people $ 2000

TOTAL $69,460

Funding source: PCR: By the end of 2015, children and mothers benefit from quality and increasingly inclusive social services for children; the country fulfils the remaining observations of the CRC on independent monitoring, data collection, resources for children and environmental health

IR: By 2015, the health system provides quality services for mothers and children in line with adopted legislative and normative frameworks aligned with international standards Indicators Grant: SC 120407 XIV Remarks and reservations

UNICEF reserves the right to withhold all or a portion of payment if performance is unsatisfactory, if work/outputs are incomplete, not delivered or for failure to meet deadlines.

All materials developed will remain the copyright of UNICEF and that UNICEF will be free to adapt and modify them in the future. Evaluators are responsible for their performance and their product(s). UNICEF reserves the copyrights and the products cannot be published or disseminated without prior written permission of UNICEF.

Candidates interested in the consultancy should submit a proposal with approximate methodological proposal, estimated cost, time line, and resume of the evaluators who will take part in evaluation process.

In order to access the Annexes the mentioned in the present Terms of Reference, please contact Tanzilya Shirvanova at tshirvanova@unicef.org. Proposals should also be submitted to the same email address.

Prepared by Nargiz Shamilova TL IMCHS Project

Reviewed by Svetlana Stefanet Chief of Health Section Zokir Nazarov M&E Officer

Approved by Berina Arslanagic-Ibisevic Deputy Representative