Results and Impact Management System (RIMS)

Devesh Roy and IFPRI/IFAD team (March 10, 2014)

Delhi

2

Management Information System (MIS)

• System -provides information at any point in the life of the project –both past and present, or up-to date information for taking projectinitiatives, and for corrective actions.

• Information in the MIS comprise both M&E information, andinformation regarding the context of the project area-critical foreffective operation of the project.

3

Basic Components of the MIS (Other elements used to the extent employed in M&E)

• Basic Statistics

• M&E Information based on Component wise Results Indicators

• Financial Information System

• Information on Production and Price

• Information on Public and Private Service Providers

• Information on Markets

• Information on Training Institutes and Research Stations

• Information on Innovations and Technologies

• Information on Wholesalers, Exporters and Transporters

• Process Documentation, etc…

Core philosophy and rationale of M&E

• Shift focus from activities and outputs to higher level results (outcomes and impact)

5

Broad steps for setting up the M&E system

• M&E stakeholders workshop for review and revision of the Logframe

• Preparation of the M&E Plan

• Train staff on RIMS and Baseline Survey

• Conduct RIMS and Baseline Survey

• Prepare the Results Based AWPB (Annual work plan and Budgeting)

• Prepare and test Input-Output monitoring forms

• Ensure that the M&E system is manually up and running & capable of generating reports

• Establish data base for other information areas for the entire MIS

• Computerize the system and test the system

• Enter data regularly, generate reports and use it for learning and decision-making

Key Steps in Setting Up an M&E System and Preparing the M&E Plan

1. Establish the Purpose and Scope;

2. Identify performance questions, stakeholders’ information needs, and indicators;

3. Define information collection tools, methodologies, frequencies, and responsibilities;

4. Planning for data processing and analysis;

5. Planning for quality communication, information sharing, and reporting arrangements;

6. Planning for critical reflection and learning events;

Key Elements of an M&E Plan (1)

• Purpose & Scope• Project overview and objectives • Rationale for M&E system design• How the system will support project management and its

partners• How the system will meet reporting requirements• Discussion of extent of participation of stakeholders

• Approach• Overview of how stakeholders will be involved (project

team members at various level, implementing partners, beneficiaries, external resources…)

• What information gathering and analyzing methods will be used

Key Elements of an M&E Plan (2)

• Performance questions, indicators, and related information needs• Define key questions to be answered to understand

whether the project is performing as planned and moving towards desired objectives;

• Definition of target outcomes, indicators and related information/data needs

• Baseline data requirement

• Assessment of stakeholders' information needs

Key Elements of an M&E Plan (4)

• Information Dissemination and Utilization• Reporting (format, frequency, responsibilities)• Communication strategy• Events for performance review and critical reflections to inform

decision making• How will the project follow up on M&E information and corrective

action• Arrangements to support and provide information to external reviews

and evaluations (supervision missions, mid-term review/evaluation, interim evaluation, completion assessment)

• M&E Work Plan and timing of Activities • Monitoring of resources, activities an implementation for effective

operations

• Evaluation of outcomes and impact for guiding the project strategy

Key Elements of an M&E Plan (5)• Establishment of Necessary Conditions and Capacities

• M&E organization • Human resource needs: staff, incentives, external assistance required • Capacity building plan for M&E: training, exposure, piloting• Resource needs: equipment, vehicles, TA….

• M&E detailed budget

• Appendices• Original and revised log frames • List of proposed indicators • Outline formats for data collection• Outline formats for reporting• Baseline survey questionnaire• M&E work plan

11

Monitoring & Evaluation ?

• Monitoring -continuous observation, reflection, and correction of activities. Done frequently –at short time intervals

• Evaluation a similar exercise, where observations on CHANGE as a result of project intervention (intended, unintended, positive and negative) is compared over time (baseline, mid-term and end-of-project), here lessons and best practices are also drawn to improve present as well as future project design and management.

Monitoring and evaluation: Why?

Support project management with information needed for decision making (management tool);

Provide information on project progress, results, and impacts;

Timely trend detection can prevent project failures and facilitate necessary adjustment to project activities and strategy;

Incorporating “lessons learned” will facilitate improving implementation and enhance chances of achieving desired impact (learning tool);

Accountability for donor funds.

13

M&E Tools in IFAD Projects

• Logical Framework

• Results and Impact Management System

• Results Based Annual Work Plan and Budget

• Annual Outcome Surveys

•Multidimensional Poverty Assessment Tool (MPAT)

RIMS: Aims and Objectives

Motivation/objective: Establish a comprehensive system for measuring and reporting on the results and impact of IFAD supported country programmes

• Provides framework for systematic reporting by the projects to IFAD and by IFAD to its governing bodies.

• Includes a menu of indicators used to measure and report on the performance of IFAD projects – at activity, output and impact level.

• The menu provides a standardized list of indicators that could be applied across all projects, and therefore can be aggregated.

• Standardized questionnaire- issue of flexibility in design and analysis??

Reporting in the context of RIMS

• More attention to impact management

• Core indicators, mandatory for all IFAD projects

• Selection of programme specific indicators from the RIMS universe

• Data on indicators reported to IFAD annually, periodically reviewed with IFAD HQ, ICO or supervision teams

• Impact surveys required at baseline and completion

Variables measured through RIMS impact surveys

The “anchor” indicators:

• Household asset index

• Child malnutrition

• Female/ male literacy

• Access to safe water

• Access to improved sanitation

Others:

• Food insecurity (intensity and spread of hungry seasons)

17

RIMS Impact Indicators

MDG IndicatorsGoal-1: Eradicate extreme poverty and

hunger

•Households with improvement in household

assets ownership index*

•Reduction in the prevalence of child

malnutrition*

Goal 2: Achieve universal primary

education

Number of Children completing

primary/secondary school (male/female)

Goal 3: Promote gender equality and

empower women

•Ratio of literate females to males

•Literacy rate (by gender)

Goal 6: Combat HIV/AIDS, malaria and

other diseases

Reduction in the incidence of infectious disease

(HIV/AIDS, malaria, tuberculosis)

Goal 7: Ensure environmental

sustainability

•People with access to improved sanitation

•People with sustainable access to an improved

source of water (drinking water)

From results to impact

1st LEVEL

RESULTS

[OUTPUTS]

2nd LEVEL

RESULTS

[OUTCOMES]

IMPACT

Measures financial and physical progress indicators

— outputs

Assesses effectiveness and sustainability and/or modified behaviour

— outcomes

Measures combined effect of 1st and 2nd level indicators

— goal

RIMS Reporting Impact Surveys

Executing RIMS from start-up to project completion

• Identify and agree on a list of indicators, including targets.

• Ensure that selected RIMS indicators are part of the M&E system.

• Monitor the indicators and report to IFAD annually.

• An annual record is to be submitted to IFAD with annual Progress reports and reviewed by IFAD supervision missions.

• Conduct Baseline assessment at start-up and impact assessment at mid-term and completion.

• RIMS Annual Outcome Surveys

Launching and implementing RIMS

• Project Logframe -reviewed to reflect RIMS indicators.

• IFAD and PMU to agree on a number of indicators for the 1st and 2nd level (along with appraisal targets and AWPB estimates)

• An annual record to be submitted to IFAD with annual Progress report and reviewed by IFAD supervision missions.

Executing RIMS: survey softwareAvailable in English, French, Spanish, Arabic

Reports:All standard RIMS analyses embedded in software; Reports produced in tables and graphs; Child malnutrition measured against new WHO standards;Relative wealth reports conducted; All reports and all data can be exported to different

formats; Follow up surveys can be compared Manual available online

22

Objective of M & E

To support managing for project results by focusing on improvement of projectperformance by

• Tracking progress of project outcome (intermediate impacts) and Impactbased on data collected at baseline, mid-term and at the end-of-project;

• Quarterly, half-yearly and annual monitoring of achievement of projectoutputs through the conversion of input to output against AWPB targets; and

• Reflecting, learning and taking actions to improve performance to achieveresults.

Key processes in the M&E system

Background on RIMS: Some salient points • Where suitable, household-centered indicators introduced to better measure project outreach.

• Rating-based approach for reporting 2nd level results. The ratings supported by a flexible mix of evidence using qualitative and/or quantitative methods including RIMS 2nd level indicators.

• Projects to choose the most suitable method for measuring 2nd level results based on local context and characteristics, including that of countries’ existing M&E systems.

• RIMS overall rationale, the guiding principles, features and concepts.

• how RIMS indicators should be selected.

• guidance on each first-level indicator including definitions and operational aspects related to reporting. methodological issues that are common to several indicators.

• Guidance in formulating assessments of second level results (and includes RIMS second-level indicators where appropriate).

• the reporting process, including sample report forms.

Measurement method

• New measurement approach for 2nd level results introduced.

• Achievement of 2nd level results measured by a rating scale. A score of 1 corresponds to highly unsatisfactory effectiveness or very weak sustainability. By contrast, a score of 6 corresponds to highly satisfactory effectiveness or very strong sustainability.

• Question to ask about the tradeoff between effectiveness and sustainability (Union or intersection of the two markers)

Measurement issues

• Question- effectiveness and sustainability- union or intersection of the two

• How does a moderate ranking (say 4) on sustainability or like it follow? Rankings are ordinal or cardinal?

• Political economy internalized in sustainability?

• How to ensure objectivity in both effectiveness and sustainability?

Effectiveness and sustainability

Overview of RIMS First- and Second-Level Results

• Natural resources (land and water)

• Agricultural technologies and production

• Rural financial services

• Markets

• Rural enterprise development and employment

• Policy and community programming

• Social infrastructure

• Total outreach

Use of RIMS level 1 and 2

• Comparing targets of 1st level results with actual achievement may lead to an analysis of factors that positively or negatively affected project implementation.

• The analysis of 2nd level results provides more in-depth information on the changes occurred at the level of beneficiaries, households, communities or institutions. • E.g. After a training program on livestock veterinary practices, the small number of

trainees adopting the recommended techniques may highlight problems with the training method like language problems or inaccessible locations.

• Taking into consideration these findings, project management may decide on training in a different form or that another training service provider will be recruited.

RIMS indicator levels

• 1st level: Project outputsReported annually

• 2nd level: outcomesReported from mid-term onwards

• 3rd level: impactAt baseline and completion:

- Asset ownership

- Child nutrition

- Water, sanitation, housing, etc.

Definitions: First-Level Results (Outputs)

• Measure financial and physical progress, mostly quantitative(numbers and percentages) and are generally planned and implemented on annual basis.

• Measures of results at either the activity or output level of the logframe.

Definitions: Second-Level Results (outputs/outcomes)

• Measure improved functionality and/or behavioural change. They are more qualitative and normally take longer to realise. Should they be or there is scope or reason to alter this???

• This level corresponds to either the output or objective level of the Project Logframe.

Definitions: Third-Level Results (impact)

• Measure the combined effect of the 1st and 2nd level results.

• quantitative and measured at specific point during project life.

• may refer to the goal level of the project logframe and in some case to the objective level.

Frequently Asked Questions

• Does the list of first-level indicators replace the previous one?

• Yes. Compared to earlier few indicators have been added and the presentation has been reorganized. For example, “people trained” has been replaced by people trained in natural resources management, people trained in infrastructure management) or the training recipient (staff of service providers)

• What if RIMS first-level indicators do not cover all activities and outputs that are relevant for my project?

• The list of 1st level indicators cannot be comprehensive of all activities and outputs of all projects. Choose only first-level indicators that are relevant to the project characteristics.

• Are there any compulsory first-level indicators?

• Yes. At least one indicator that shows the number of individuals and households that during the period under review have received project services should be reported.

• Do all indicators under a given category need to be reported?

• No, the indicators have been categorized for ease of reference. For a given component, relevant indicators may be found in any of the categories used in the Handbook.

• Should first level indicators continue to be reported on the RIMS form if they have been dropped?

• No, but the data collected may prove useful for the project M&E.

• What if there are no planned or appraisal target figures?

• If targets are not available, NA should be reported. However, planning for results is necessary for project accountability and is an important element of the M&E system. The Logframe may need to be revised so that indicators and appraisal targets are established.

36

The Logframe Matrix

• Used to communicate key information about

project objectives, outcomes, outputs and

activities in a systematic and logical way.

• The basic Logframe matrix contains 4 columns

and 4 rows.

• Describes external factors that influence the

project’s success: assumptions and risks

37

The Logical Framework Matrix

Objectives &

activities

Purpose/

(Outcome)

Goal

(Impact)

Outputs

ActivitiesMeans

IndicatorsMeans of

verificationAssumptions

Cost

Key Elements of an M&E Plan (3)

• Data/Information Sources, Gathering Methods, and Responsibilities• How, who, when of data collection

• Data/Information Management Information System (MIS)• How, who, when of data storage, synthesis, and

aggregation

• Data/Information Processing and Analysis • How, who, when of data analysis

FAQs: Continued• Why is achievement of second-level results reported with a rating?

• Ratings introduced to allow projects to report achievement of 2nd level results by using project-specific indicators and any other evidence available. Assessments allow information from other sources, in particular national systems. Ratings also align RIMSwith other evaluative instruments such as Project Completion Reports and formal evaluations.

• How about the risk that rating may be too subjective?

• Rating should be a realistic and candid assessment by as wide a group of stakeholders and may be validated during supervisionmissions and project reviews.

• Who should rate the 2nd level results?

• Unless IFAD and project agree on a different arrangement, the rating of 2nd level results should be done by the project.

• When should second-level results be reported?

• 2nd level results should be reported after the Mid-Term Review. The rating should be updated annually based on new evidence. 2nd level results can be reported earlier if evidence is available.

• What are the implications of a low rating for a second-level result?

• A low rating does not imply that the project management is not committed to achieve the development objectives. Lower-than-expected results should be used to identify corrective actions

• What if the M&E system does not include indicators for assessing RIMS second-level results?

• Outcome-level indicators need to be included in the project M&E system. Consider reviewing the project Logframe in the next annual planning phase so that appropriate indicators can be included

Qualitative tools

1. Key Performance Questions

2. Knowledge, Attitude, Practice (KAP) Surveys

3. Most Significant Change (MSC)

4. Focus Group Discussions

5. Key Informant Interviews

FAQs: continued

• How is the supporting evidence for second level results submitted?• When the RIMS form is submitted (normally an Excel file), the supporting evidence used

for assessing 2nd level results should be included in an additional worksheet or separate text document(s).

• Clear indications of the methodology and source should be provided. • The supporting evidence should be in summary form. • Where studies or surveys have been carried, it is not necessary to submit the entire

study/survey but clearly reference these as source material. • The evidence may include RIMS second level indicators which should be reported as

evidence under the relevant second level result.• When should the RIMS form be sent to IFAD?• The deadline for submission of the RIMS form is 31 March.• To what period do RIMS results refer?• RIMS results refer to the period of the Annual Work Plan and Budget (e.g. January to

December or July to June).

Challenges (from IFAD perspective)

Data quality problems:

Problems of double-counting of outputs

Data not produced timely (e.g. before supervision missions)

Few credible data on outcomes and impact

Challenges and risks

RIMS surveys methodology not really adequate to measure project impact:

No control/comparison group

Sample households are not necessarily project beneficiaries

Food insecurity and child malnutrition measures highly sensible to annual variations in food production

Three malnutrition variables difficult to interpret

RIMS First and Second level reporting

• First section of first level results

• First level results are reported annually

• AWPB target and actual result-km of roads target to be constructed/rehabilitated and actual km of roads completed/rehabilitated – Percentage of target met

• Second section of first level results

• Cumulative value of each indicator and comparison with the target results reported in the Logframe or in the Appraisal documents.

Illustration

Reporting of RIMS second level results• Recall - Second-level results is reported on the basis of a rating scale from 1 to 6. A

score of 1 corresponds to highly unsatisfactory effectiveness or very weak sustainability. A rating of 6 corresponds to highly satisfactory effectiveness or very strong sustainability.

• The rating should be reported under the column of “actual” for the period under review. No other value or number should be shown in the RIMS form for second-level results. The section on cumulative and appraisal values does not apply to RIMS second-level results.

• When the RIMS form is submitted (normally an Excel workbook), supporting evidence used for the assessments should be included in an additional worksheet or separate text document(s).

• Clear indications of the methodology and source should be provided. The supporting evidence should be in summary form.

• RIMS second-level results should be provided at the time of Mid-term Review (MTR). The ratings of RIMS second-level results should be updated every year on the basis of the new data and evidence available.

Checklist for sending the RIMS form • Clearly outlined in the RIMS manual - reminder about gold medal, silver medal and bronze

medal gaps

• Are first-level indicators reported in the appropriate units (number, hectares, kilometers, etc.) as specified in the indicator charts?

• Where relevant, are sex-disaggregated figures reported or other stratification?

• For projects after PY2, are cumulative figures consistent with the data in the previous years? Have figures for previous years been attached or entered on the spreadsheet?

• If revisions have been undertaken in the Logframe, are targets updated?

• Are the ratings for second-level results shown? Are ratings updated on the basis of the new evidence and data available?

• Are papers with evidence and supporting indicators of second-level results attached?

Challenges in reporting

• Your views!!!

• Is the format demanding – what project specific changes you would recommend

• Is there an agency problem in reporting?