Using Indicators to Measure Progress and Performance

Using Indicators to Measure Progress and Performance


1

Dominique Senn, seecon gmbh


Find this presentation and more on www.sswm.info

Copy it, adapt it, use it – but acknowledge the source!CopyrightIncluded in the SSWM Toolbox are materials from various organisations and sources. Those materials are open source. Following the open-source concept for capacity building and non-profit use, copying and adapting is allowed provided proper acknowledgement of the source is made (see below). The publication of these materials in the SSWM Toolbox does not alter any existing copyrights. Material published in the SSWM Toolbox for the first time follows the same open-source concept, with all rights remaining with the original authors or producing organisations.

To view an official copy of the the Creative Commons Attribution Works 3.0 Unported License we build upon, visit http://creativecommons.org/licenses/by/3.0. This agreement officially states that:You are free to:

• Share - to copy, distribute and transmit this document • Remix - to adapt this document. We would appreciate receiving a copy of any changes that you have made to improve this

document. Under the following conditions:

• Attribution: You must always give the original authors or publishing agencies credit for the document or picture you are using.

DisclaimerThe contents of the SSWM Toolbox reflect the opinions of the respective authors and not necessarily the official opinion of the funding or supporting partner organisations. Depending on the initial situations and respective local circumstances, there is no guarantee that single measures described in the toolbox will make the local water and sanitation system more sustainable. The main aim of the SSWM Toolbox is to be a reference tool to provide ideas for improving the local water and sanitation situation in a sustainable manner. Results depend largely on the respective situation and the implementation and combination of the measures described. An in-depth analysis of respective advantages and disadvantages and the suitability of the measure is necessary in every single case. We do not assume any responsibility for and make no warranty with respect to the results that may be obtained from the use of the information provided.

Copyright & Disclaimer

http://www.sswm.info/

http://creativecommons.org/licenses/by/3.0



Contents

1. Introduction2. What Are Indicators3. The Selection of Indicators4. Why Use Indicators5. What to Use Indicators for6. The Development of Indicators7. Typical Indicators in SSWM8. Advantages and Disadvantages9. References

3




Every project and programme has specific objectives. In order to assess the progress and performance of the project (e.g. to analyse whether targets are met, to give account of the progress to donor agencies or to show accountability to the public, funders, donors and stakeholder groups), monitoring and evaluations need to be carried out.

4

1. Introduction

Source: http://www.clearhorizon.com.au/wp-content/uploads/2009/02/theproject25percent.jpg [Accessed: 23.10.2013]

Indicators provide an effective tool for measuring progress and performance.




Definitions•OECD/DAC: “A quantitative or qualitative factor or variable that

provides a simple and reliable means to measure achievement, to reflect changes connected to an intervention, or to help assess the performance of a development actor.”

•USAID: “A variable, which purpose it is to measure change in a phenomena or process.”

•UKAID & United States Institute of Peace: An Indicator “refers to a measure used to demonstrate change in a situation, or the progress in, or results of, an activity, project, or program.”

•European Commission: “A description of the project’s objectives in terms of quantity, quality, target group(s), time and place.”

Source: MDF 2005, UKAID & UNITED STATES INSTITUTE FOR PEACE n.y.

5

2. What Are Indicators




Example

Specific indicators have been developed in order to measure progress towards the Millennium Development Goals.

For the full list, see: http://mdgs.un.org/unsd/mdg/Host.aspx?Content=Indicators/OfficialList.htm

6

2. What Are Indicators

Goals and Targets Indicators

Goal 1: Eradicate extreme poverty and hungerTarget 1.A: Halve, between 1990 and 2015, the proportion of people whose income is less than one dollar a day

1. Proportion of population below $1 (PPP) per day2. Poverty gap ratio 3. Share of poorest quintile in national consumption

Target 1.B: Achieve full and productive employment and decent work for all, including women and young people

1. Growth rate of GDP per person employed2. Employment-to-population ratio3. Proportion of employed people living below $1 (PPP) per day4. Proportion of own-account and contributing family workers in total

employment Target 1.C: Halve, between 1990 and 2015, the proportion ofpeople who suffer from hunger

1. Prevalence of underweight children under-five years of age2. Proportion of population below minimum level of dietary energy

consumption

Goal 2: Achieve universal primary educationTarget 2.A: Ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling

1. Net enrolment ratio in primary education2. Proportion of pupils starting grade 1 who reach last grade of primary 3. Literacy rate of 15-24 year-olds, women and men


http://mdgs.un.org/unsd/mdg/Host.aspx?Content=Indicators/OfficialList.htm



(1/2)

Generally an indicator focuses on a small, manageable set of information that gives a sense of the bigger picture. The choice of indicators is important as to whether it gives sufficient “sense of the bigger picture”. Only well selected indicators can assist in maintaining a focus on the important work areas and take strategic decisions to address problem areas. The careful selection of indicators makes a project sustainable and allows responsible persons to act.

7

3. The Selection of Indicators




(2/2)

The term “minimum indicators” is used in recognition that:•It is better to start with a small set of indicators that are

feasible to monitor and to improve over time.•There are many other indicators that could be used to

measure progress beyond this basic level.

8

3. The Selection of Indicators




An indicator is used to answer the question: How do we know if and how much we have achieved our objective?

Indicators are useful to:•Measure progress and performance over time against various water and sanitation objectives to evaluate the effect of policy actions and plans. •Provide information on the status, trend or performance of a given system to inform the project team (responsible organisation), stakeholders, the public and policy makers in a simplified way. •Translate data into policy relevant information (describe, show trends, communicate results of implementing objectives).•Identify areas for increased attention by an organisation.

9

4. Why Use Indicators




Criteria for Developing IndicatorsGood indicators should meet certain criteria. Three different concepts are presented on the following slides: SMART indicators, SPICED indicators and CREAM indicators. While SMART describes the properties of the indicators themselves, SPICED relates more to how indicators should be used.

But keep in mind: the most important point in developing indicators is:Don’t be too ambitious! Start with what can be realistically done or else failure is guaranteed.

10

6. The Development of Indicators

!




SMART Indicators

11


S SPECIFICThe indicators need to be clearly articulated, well defined and focused. They should be clear to people with a basic knowledge of the issue, programme or initiative.

M MEASURABLE

The indicator should be measurable, that is, it has the capacity to be counted, observed, analysed, tested, or challenged. The indicators should be able to determine the degree of completion or attainment. Using the same methodology and information, findings should be repeatable and comparable.

A ATTAINABLEThe target attached to the indicator should be achievable within the scopes of the project/ programme.

R RELEVANTThe indicators should be able to detect change and be related to the specific situation they are “indicating” information about. They should be measurable at an appropriate scale.

T TIME-BOUND The indicator should be attached to a time frame. Term dates for measurement should be included. Source: Save the Children, https://sites.google.com/site/savethechildrendme/Home/smart-indicators [Accessed: 17.12.2013]


https://sites.google.com/site/savethechildrendme/Home/smart-indicators



S SUBJECTIVEInformants have a special position or experience that gives them unique insights which may yield a very high return on the investigators time. In this sense, what may be seen by others as 'anecdotal' becomes critical data because of the source’s value.

P PARTICIPATORY

Indicators should be developed together with those best placed to assess them. This means involving a project's ultimate beneficiaries, but it can also mean involving local staff and other stakeholders.

IINTERPRETED & COMMUNICABLE

Locally defined indicators may not mean much to other stakeholders, so they often need to be explained.

C CROSS-CHECKED & COMPARED

The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers.

E EMPOWERINGThe process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation.

D DIVERSE & DISAGGREGATED

There should be a deliberate effort to seek out different indicators from a range of groups, especially men and women. This information needs to be recorded in such a way that these differences can be assessed over time.

SPICED Indicators

12


Source: MDF 2005




Indicators should be...CREAM Indicators

13


C CLEAR ...Precise and unambiguous

R RELEVANT ...Appropriate to the subject at hand

E ECONOMIC ...Available at reasonable cost

A ADEQUATE ...Able to provide sufficient basis to assess performance

M MONITORABLE ...Amenable to independent validation

Source: UKAID & UNITED STATES INSTITUTE FOR PEACE n.y.




(Adapted from MUGA and MIHELCIC 2008)

•Functional indicators (e.g. adaptability, durability and reliability of the system)

•Economic indicators (e.g. capital costs, operation and maintenance costs and user costs)

•Environmental indicators covering resource use (e.g. nutrient reuse) and emissions (e.g. emissions of carbon dioxide)

•Social-cultural indicators (e.g. public participation, acceptance, and stimulation of sustainable behaviour)

14

7. Typical Indicators in SSWM




Examples of Indicators for Water Sources Management•Water withdrawal: absolute or per-person value of yearly

water withdrawal gives a measure of the importance of water in the country’s economy.

•Ground water level development (rise or fall of groundwater levels, pollution).

•Water sources quality (e.g. turbidity, minerals content, pathogens, chemical quality).

15


Water quality assessment with different indicators (biodiversity, chemical substances, nutrients, toxics, sediments, etc.).

Source: http://www.epa.gov/region1/lab/reportcardintro.html [Accessed: 10.12.2013]


http://www.epa.gov/region1/lab/reportcardintro.html



Examples of Indicators for Water Purification

•Treatment efficiency indicator (e.g. removal of indicator bacteria)

16


Removal efficiency of organic nitrogen in different treatment methods (1-nitrification, 2-oxidation pond, 3-chemical coagulation, 4-chlorination, 5-ammonia removal, 6-filtration, 7-reverse osmosis).

Source: http://www.intechopen.com/books/water-treatment/relationship-of-algae-to-water-pollution-and-waste-water-treatment [Accessed: 10.12.2013]


http://www.intechopen.com/books/water-treatment/relationship-of-algae-to-water-pollution-and-waste-water-treatment





Examples of Indicators for Water Use•Percentage or number of people not served with improved drinking

water and extension of piped water supply.

•Domestic water use efficiency (per capita use of water).•Agricultural water use efficiency and efficacy (water demand per

irrigated area).•Industrial water use.•Awareness and actual use of water-saving appliances.

17


Number of household connections per 100 residents in urban sub-Saharan Africa.Source: WSP & WUP 2003




Examples of Indicators for Wastewater Collection

•Percentage of wastewater collected to total wastewater production.

18


Particularly in peri-urban and rural areas in developing countries, wastewater is rarely collected, representing high health risks to the people living in these aras. Source: http://www.ens-newswire.com/ens/aug2007/2007-08-07-01.asp


http://www.ens-newswire.com/ens/aug2007/2007-08-07-01.asp

http://www.ens-newswire.com/ens/aug2007/2007-08-07-01.asp



Examples of Indicators for Wastewater Treatment•Percentage or number of people not served with basic sanitation.

•Number of people affected by water borne diseases (e.g. diarrhoea, cholera).

•Investment in drinking water supply and sanitation.•Removal of pollutants.•Awareness and actual use of safe sanitation systems.

19


Percentage of households having a sewerage connection in sub-Saharan African cities.Source: WSP & WUP 2003




Examples of Indicators for Reuse and Recharge of Nutrients and Water

•Percentage of reclaimed water used in agriculture to total agricultural water use.

•Percentage of nutrients used from recycled sources.

20


Proportion of recycled water to withdrawn water.

Source: http://www.potashcorp.com/annual_reports/2012/gri/environment/#EN8 [Accessed: 10.12.2013]


http://www.potashcorp.com/annual_reports/2012/gri/environment/#EN8

http://www.potashcorp.com/annual_reports/2012/gri/environment/#EN8



21

8. Advantages and Disadvantages

•Failures can be detected and eliminated.

•Indicators can be adapted to the circumstances.

•It is not necessary to measure every single parameter to get a “bigger picture” over the whole progress.

•A set with too many indicators will tend to clutter the overview.

•Measuring indicators is a long-term exercise. If the indicators are not defined properly, and if measuring is not done correctly and conscientiously, it will not work.




22

Business Dictionary. URL: http://www.businessdictionary.com/definition/indicator.html#ixzz2mVtidEG6 [Accessed: 04.12.2013]

MDF (Editor) (2005): MDF Tool: Indicators. Ede: MDF Training and Consultancy. URL: http://www.undg.org/docs/11652/MDF-Indicators-Brief-%282005%29.pdf [Accessed: 04.12.2013]

MUGA, H.E.; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management 88, 437 - 447. URL: http://www.ncbi.nlm.nih.gov/pubmed/17467148 [Accessed: 23.10.2013]

TAYLOR, P.; LIDEN, R.; NDIRANGU, W.; JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net). URL: http://www.cap-net.org/sites/cap-net.org/files/RBO%20Manual.pdf [Accessed: 23.10.2013]

UKAID & UNITED STATES INSTITUTE FOR PEACE (n.y.): Indicator Module. URL: http://dmeforpeace.org/sites/default/files/3.9%20Indicators.pdf [Accessed: 10.12.2013]

WUP and WSP (2003): Better Water and Sanitation for the Urban Poor. Good Practice from Sub-Saharan Africa. Kenya: Water Utility Partnership (WUP) and The World Bank Water and Sanitation Program (WSP). URL: http://www.wsp.org/sites/wsp.org/files/publications/330200725049_afBetterWaterandSanitationForTheUrbanPoorGoodPracticeFromSSA.pdf [Accessed: 13.09.2013]

ZEWO (Editor) (n.y.): Outcome and Impact Assessment in International Development. Zewo Guidelines for Projects and Programmes. Zurich: Schweizerische Zertifizierungsstelle fuer gemeinnuetzige, Spenden sammelnde Organisationen (ZEWO). URL: http://www.zewo.ch/impact/en/impact [Accessed: 23.10.2013]

9. References


Using Indicators to Measure Progress and Performance 23

“Linking up Sustainable Sanitation, Water Management & Agriculture”

SSWM is an initiative supported by:

Created by:

Documents

Using Indicators to Measure Progress and Performance