Using a results-based evaluation methodology in a public sector environment that focuses on...

Using a results-based evaluation methodology in a public sector environment that focuses on activities”

Terms of Reference - Example

• To conduct an evaluation of the effectiveness and impact of the DLGTA support over the past five years using a results based management (RBM) approach.

• Investigation into the performance and management of the community libraries recapitalisation programme

• “The department of economic development agency micro-level monitoring and evaluation system”.

• Evaluation of the soul buddyz clubs

What are we faced with at the commencement of the monitoring and evaluation assignment

Planning Statements:Strategic Goals - Examples

• To invest and ensure the provision of quality social welfare services to children including those in need of care and protection.

• Facilitate integrated and responsive governance in a developmental state.

• Promote health, prevent and manage illness or conditions with emphasis on poverty, lifestyle, trauma and violence and psychosocial factors

Planning StatementsStrategic Objectives - Examples Strategic Objective Indicator

Ensure that municipalities meet basic needs of communities.

Lateral contribution

• Executive development learning framework.

• Capacity for better implementation of donor funded projects

TransformationDesign and deliver innovative multimodal programmes.

Planning Statements Measurable Objectives - Examples • Reduce infant and under five morbidity and mortality

• To ensure effective leadership, management and administrative support to the department through the continuous refinement of organisational strategy and structure, in compliance with appropriate legislation and best practice.

• Conduct campaigns on key nutrition priorities such as obesity

• To promote growth, social development and poverty reduction through sound fiscal and financial policies, and the effective, efficient and appropriate allocation of public funds.

Results Statements Outcomes - ExamplesOutcome Indicators

Enhance service delivery

Regulatory and support mechanisms for Municipal Councils and ward committees reformed and implemented

Maturity in M&E system for cooperative governance

A coordinated and functional M,R&E system for Provincial and Local Government

Coordinated M&E system for cooperative governance

Regulatory framework for M&E implemented

Results Statements Outputs - Example Output indicators

Improved quality and quantity of measurable objectives and trend based performance indicators by departments andpublic entities

Publication and implementation of a standard operating procedures manual

Local government budget Framework Integrity of the framework: fiscal sustainability, structure and trends in fiscal indicators

Infrastructure delivery improvement programme

Technical Assistants deployed in targeted provincial Departments

Enhance coordination across government for effective implementation of MFMA

An agenda that ensures alignment ofactivities of national and governmentdepartments consistent with agreedPriorities

Where do we start

Where do we start• Review all planning documents – these include:

– Strategic plans– Annual performance plans– Business Plans– Operational Plans

• Map the planning statements, the indicators, targets, activities etc.

• Identify all available data – Quarterly reports– Annual Reports– Baseline data– Reports developed for special studies conducted

Evaluation ParametersResult’s MatrixResult’s Matrix

Reconstructing the Intervention Logic• Check whether the planning statements (goal, strategic objectives and measurable

objectives) could be reformulated as results statements (output, outcome and impact), so they can be measured accordingly.

• Check whether the sum of the planned components/support is sufficient to produce the intended result.

• Explicitly describe the planning assumptions and minimize the risk of failure (did we assume too much?)

• Map the indicators detailed in the Strategic Plans, APP and the Operational Plans to the remapped results matrix.

• In trying to establish the model of evaluation and the methodology to be employed for the evaluation the following should be considered:– remapping of the planning levels using the parameters governing logic models. – establish whether there are linkages between the activities, the outputs, the

outcomes and the impacts.

DIR

EC

T

OU

TC

OM

E

DIR

EC

T

OU

TC

OM

E

INTER

MED

IA

TE

OU

TC

OM

E

INTER

MED

IA

TE

OU

TC

OM

E

OUTPUTSOUTPUTS

OUTPUTSOUTPUTS

OUTPUTSOUTPUTS

OUTPUTSOUTPUTS

OUTPUTSOUTPUTS

DIR

EC

T

OU

TC

OM

E

DIR

EC

T

OU

TC

OM

E

ProjectResourcesProject

Resources

INPUTSINPUTS

INPUTSINPUTS

INPUTSINPUTS

ACTIVITIES

ACTIVITIES

INPUTSINPUTS

INPUTSINPUTS

ACTIVITIES

ACTIVITIES

ACTIVITIES DIRECT OUTCO

ME

DIRECT OUTCO

ME

INTER

MED

IA

TE

OU

TC

OM

E

INTER

MED

IA

TE

OU

TC

OM

E

ULTIM

ATE

OU

TC

OM

EU

LTIM

ATE

OU

TC

OM

ELogic Model

“Would you tell me, please, which way I ought to go from here?”

“That depends a good deal on where you want to get to,” said the Cat.

“I don’t much care where…” said Alice.

“Then it doesn’t matter which way you go,” said the Cat.

“…so long as I get SOMEWHERE,” Alice added as an explanation.

“Oh, you’re sure to do that,” said the Cat, “if you only walk long enough.”

• Evaluability assessment is a brief preliminary study to determine whether an evaluation would be useful and feasible.

• Establish the discrepancies between the rhetoric and the reality – according to Nay and Kay (1982) different levels of policy makers and programme managers may have different rhetorical models for the same programme and they cling to these models because it is perceived that approach to bring about change.

• The planning framework utilised and the evidence of the logical linkages between the different levels of planning.

• The appropriateness and plausibility of the Strategic goals and Strategic Objectives in terms of the legislative mandate and core functions of the province.

Evaluability Assessment

• Assess the extent to which the intervention logic provides evaluators with operational benchmarks against which outputs, outcomes, impacts and assumptions can be evaluated.

• Check whether necessary baseline and monitoring data is available. • An exploration of the evaluation models to determine the degree the models are relevant,

appropriate and feasible in terms of:– The availability of information – routine, ad hoc and research information– The logic used to construct the programmes– The timing and relevance of such an evaluation.

• Check the availability of key informants, such as planners, intervention staff, and target group

representatives.

• Assess the extent to which the evaluation questions can be answered, given the timing of the evaluation in relation to the current phase of the intervention cycle.

Evaluability Assessment

The Evaluation

Measuring change and inferring causality• Relevance

– The extent to which a development intervention conforms to the needs and priorities of target groups and the policies of recipient countries and donors.

• Sustainability– The continuation or longevity of benefits from a development intervention

after the cessation of development assistance.• Efficiency

– The extent to which the costs of a development intervention can be justified by its results, taking alternatives into account.

– Assumes that there has been effectiveness• Other evaluation criteria

– Appropriateness –tailored to local needs and changing demands– Coverage –evenness, comprehensiveness– Connectedness –co-operation – Coherence and alignment

Components of an Evaluation

• There are 3 general components to comprehensive evaluation:

– Process evaluation: How was the strategy/ programme implemented?

– Outcome evaluation: Did the strategy/ programme meet its objectives?

– Impact evaluation: Was the ultimate goal of the strategy/programme achieved?

Every program has… Every program evaluation should have…

Impact Indicators

Outcome Indicators

Process Indicators

GoalsGoals

ObjectivesObjectives

ActivitiesActivities

Every Operational Plan/Programme has…

Every evaluation should have…

Impact Indicators

Outcome Indicators

Process Indicators

GoalsGoals

ObjectivesObjectives

ActivitiesActivities

What is Process Evaluation?

• Process evaluation: – Addresses how, and how well, the program is functioning

• It can help to…– Create a better learning environment– Improve presentation skills– Show accountability to funder– Reflect the target populations– Track service units

Process Evaluation con’t

• Key questions in process evaluation:– Who is served? – What activities or services are provided? – Where, when, and how long is the program?

• Identify how a product or outcome is produced• Identify strengths & weaknesses of a program• Create detailed description of the program

–

Every Strategic Plan/Operational Plan/Programme has…

Every program evaluation should have…

Impact Indicators

Outcome Indicators

Process Indicators

GoalsGoals

ObjectivesObjectives

ActivitiesActivities

Outcome Evaluation

• Outcome evaluation: – Measures the extent to which a Strategy/

programme produces its intended improvements– Examines effectiveness, goal attainment and

unintended outcomes– In simple terms, “What’s different as a result of

your efforts?”

Outcome Evaluation con’t

• Key questions in outcome evaluation:– To what degree did the desired change(s) occur?

• Outcomes can be immediate, intermediate or longer-term

• Outcomes can be measured at the patient, provider, organization, or system level.

Every Strategic Plan/ Programme has…

Every evaluation should have…

Impact Indicators

Outcome Indicators

Process Indicators

GoalsGoals

ObjectivesObjectives

ActivitiesActivities

Impact Evaluation

• Impact is sometimes used to mean “ ultimate outcome.”

• Impact is perhaps better defined as a longer-term or ultimate outcome.

Results-Based Evaluation

Results-Based Evaluation is

an assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process.

31

INFORMATION (MIS) USE

Focus should be knowledge generation