Harmonization of Monitoring and Evaluation Indicators for ARV … · Harmonization of Monitoring and Evaluation Indicators for ARV Procurement and Supply Management Systems Working

`

Harmonization of Monitoring and Evaluation Indicators for ARV

Procurement and Supply Management Systems

Working Document for Field Testing

May 2008

AMD Partner Network

2

Table of Contents

Table of Contents.................................................................................................................. 2

Acknowledgement................................................................................................................. 3

1. Introduction ....................................................................................................................... 5

2. Rationale........................................................................................................................... 5

3. Purpose ............................................................................................................................ 5

4. Development process ....................................................................................................... 5

5. Target group .................................................................................................................... 6

6. Key ARV Procurement and Supply Management Issues and related indicators ................ 7

6.1 Introduction.................................................................................................................. 7

6.2 Components of the Medicine Supply Cycle and related M&E Indicators ...................... 7

a. Selection .................................................................................................................... 7

b. Prescription and Use .................................................................................................. 8

c. Forecasting ................................................................................................................ 9

d. Procurement and Ordering......................................................................................... 9

e. Quality Assurance .................................................................................................... 10

f. Inventory control ...................................................................................................... 11

g. Availability of ARV medicines ................................................................................... 12

h. Distribution ............................................................................................................... 12

i. Human Resources .................................................................................................... 13

j. Funding Sources and Budgeting ............................................................................... 14

k. Organization and Management ................................................................................ 14

l. Policies, Legislation and Regulations ........................................................................ 14

7. Annexes ………………………...………………………………..……………………….……..16

Annex 1 Core M&E ARV- PSM Indicators

Annex 2 Supplementary M&E ARV - PSM Indicators

8. References……………………...………………………………..……………………….……..30

3

Acknowledgement This document was developed with the active involvement of many national and international

institutions over the past three years. We are grateful to the following participants of the WHO consultation on Harmonization of M&E requirements for ARVs procurement and supply

management (PSM) systems , held in Geneva, 10-11 October 2005:

• Hélène Degui and Alassane Ba (CHMP), Luca Li Bassi and Steen Stottrup (Global

Fund), Xenophon Santas (PEPFAR/OGAC), Henk W.A. den Besten and Charles Chiedza Maponga (IDA Solutions), Yasmin Chandani (JSI), Douglas Keene (MSH/RPM

Plus), Bechir N'Daw (UNAIDS), Helene Moller (UNICEF), Bert Voetberg (World Bank),

Wilbert Bannenberg (Consultant), Andrew Loke (Pharmaniaga Solutions), Ernest Rwagasana (CAMERWA), Martin Olowo Oteba (MOH Uganda), Nguyen Van Kinh (MOH

Vietnam), Cao Thi Than Thuy (WHO Office - Vietnam), Emma Nelson Msuya (National

AIDS Control Programme – Tanzania), Deus Bazira Mubangizi (Catholic Relief Services), Carole Presern (UK Mission), and the following WHO staff: Teguest Guerma,

Jos Perriëns, Mary Couper, George Loth, Amolo Okero, Vincent Habiyambere, Cyril

Pervilhac, Helen Tata, Jeff Sinden and Marco Vitoria.

We specifically would also like to thank:

• the participants and facilitators of the IDA Solutions training workshops held in 2006,

2007 and 2008 on M&E of ARVs PSM systems. They provided valuable input on the indicators which were reduced and categorized into 11 core indicators and 30

supplementary indicators: Rumen Andreev, Viviane Leu, Hailu Tadeg, Abiy

Andargachew, Akaki Lochoshvili, Naana Frempong, Sookdeo Singh, Sharmini Khalikaprasad, Konan Jules Yao, Clement Kouakou, Matebele Sefali, Masoko Nts’ekhe,

Lerato Kholokholo, Doris Tshabalala, Remi Olaitan, Liliana Caraulan, Eric Nyiligira, Ivan

Makumbi, Vladislav Volchkov, Bamikale Feyisetan, Thomas Wushe, Daniel Taddesse,

Yohannes Tsegay Berhe, Mulusew Lijalem Belew, Abdulmalik Hassen Ebro, Nesrane Senbet Sahlemariam, Mehamed Feleke Tessema, Jurgen Hulst, Alemu Shiferaw

Damassa, Grace Waiharo, James Batuka, Laurentiu Ionesii, Sushil Kumar Koirala, Ivana

Lohar, Ishaya Dawha, Loic Aubry, Galina Bolshakova, Ludmila Reutskaya, Tifenn Humbert, Karin Wiedenmayer, Angela Taylor, Aminu Abubakar, Yakubu Adamu,

Kgosiemang Atamelang Sidney, Olivia du Moulin, Elfatih Elamin, Amjad Idries, Jennie

Lates, Okumu Morris, Mohamed Motwaly, Frederick Nicolaus Rubanga, Gashaw

Shiferaw Mekonnen, Hendro Supaat, Jansen Susanna, Seyoum Temenit, Araia Berhane, Tesfaselase Ghedel, Habte Desbele, Elina Sverdlova, Albert Ntiringaniza, Be

El Hassan, Félicienne Trévant, Jenny Jasmine Akondja Yandja, Carl François, Jean

Claude Lundu Deka, Michèle Razanamparany, Tantely Rakotomalala, Marthe Everard, Jean-François Saint-Sauveur, Chiedza Maponga, Peter Graaff, Vincent Habiyambere,

Cees Tuinenburg and Clarisse Morris.

• The members of the JSI/MSH/SCMS working group (Dana Aronovich, Timothy Williams,

Yasmin Chandani, Youssouf Ouedraogo, Naomi Printz [all from JSI], Bannet

Ndyanabangi, Helena Walkoviak, Leila Akhlaghi [all from MSH] and Gary Bettger,

Jennifer Mboyane, Niall Shanbhag and Sarah Levine [all SCMS].

• The following people who, during the French adaptation process allowed to improve,

both the English and the French versions: Carinne Bruneton and Charles Rambert from ReMeD and Caroline Damour from ESTHER.

• Daisy Carandang, Dennis Ross-Degnan, John Chalker, Cyril Pervilhac, Yves Souteyrand, Chika Hayashi, Dongbau Yu, Camellia Falcon, Phillip Savio and Priya

Emmart.

4

This document has been written by the following persons to whom WHO extends its

acknowledgement: Henk den Besten, Cees Tuinenburg, Clarisse Morris, Jos Perriens, Peter Graaff and Vincent Habiyambere. The present version is a working document ready for field

testing. Any comment to improve the next version should be sent to Dr Vincent Habiyambere

at [email protected].

5

1. Introduction

Tracking key aspects of procurement and supply management (PSM) and taking corrective

action when required is a means to continuously improve the effectiveness of a programme

or a system. This monitoring should cover different components of the PSM system. It must be noted that just monitoring does not improve the PSM system. M&E shows the strengths

and the weaknesses of the system. If the measurements are below the target this should

trigger the responsible person to investigate and correct the problem. This is how monitoring can assist in ongoing quality assurance of the PSM system. Trained human resources in

sufficient number, financial and other resources for the effective implementation of the M&E

system should be available. 5-10% of the programme costs are recommended to be

allocated to M&E activities.

2. Rationale

Monitoring and Evaluation (M&E) can be used for two main purposes.

Firstly, M&E is used to regularly check and evaluate the processes and the results of a programme and to find out whether progress is being made towards the objectives. When

M&E detects that the performance of any programme component is below the expectations,

actions to prevent and/or to correct the problems should be initiated. Used this way, M&E is

a tool for continuous improvement of a programme’s performance while at the same time facilitating reporting, accountability and transparency.

Secondly, M&E is used to inform donors on the programme’s effectiveness and efficiency. Donors require information that justifies how the funds were spent and what were the

achievements. M&E is also used to compare programmes in an objective manner.

Programmes funded by various donors have specific objectives, which call for specific indicators. Indicators to measure the performance of various programmes may be different

given the specificity of each programme and given the requirements of each donor.

However, indicators to monitor the national PSM system should be common for all PSM

programmes at country level.

3. Purpose

The purpose of this document is to harmonize M&E indicators to measure the effectiveness

and the performance of the national PSM system: national M&E system and donors are

encouraged to use the same indicators to avoid dispersion of efforts on the M&E agenda.

This is in line with the Three Ones principle. As much as possible, preference will be given to indicators that can be measured using information that is already being collected or could be

collected on a routine basis so that the burden on human resources is limited.

4. Development process

In 2004 the AMD Partner Network and the AMD Secretariat in WHO/HIV Department recognized the need for the different donors funding HIV/AIDS programmes to harmonize

their reporting requirements at country level in order to increase transparency, productivity

and efficiency in various programmes. Since then WHO/AMDS consulted IDA Solutions to

perform a desk study on reporting requirements including M&E indicators for ARV procurement and supply management.

The results of the desk research were discussed at the harmonization meeting organized by WHO/AMDS in October 2005. Participants included representatives from central medical

stores, ministries of health, WHO regional offices, and several international donors. Some of

these organizations submitted their M&E indicators for different PSM aspects.

6

Following the meeting, JSI and MSH as part of SCMS, were requested to compile key PSM indicators which could be used for continuous programme tracking as well as for donor

reporting. This working group produced the initial list of 50 indicators which was used as a

basis for further development and simplification

In June 2006 and March 2007 the draft M&E indicators for ARV PSM system were

discussed during IDA Solutions training sessions on Monitoring & Evaluation of Antiretroviral

Procurement & Supply Management Systems. This training was attended by health staff involved in M&E and in ARV procurement and supply management for low and middle

income countries. The participants and facilitators reviewed the 50 original indicators and

made valuable suggestions on deletion of a few items and on selection of priority indicators based on their experience in the field.

WHO, guided by the above input and feedback from other experts, finalized a list of 11 Core

Indicators. After deletion of 9 items, the remaining 30 non-prioritized indicators, here called "Supplementary Indicators" can be measured if more information is needed. Core and

Supplementary Indicators are found in Annexes 1 and 2 respectively.

5. Target group

These indicators provide a practical tool for staff in charge of planning, management,

implementation, monitoring and reporting on national PSM systems. This document can also

be used by institutions and donors who need to monitor the performance of the PSM

programmes for which they are providing technical and/or financial support.

6. Key ARV Procurement and Supply Management Issues and Related M&E Indicators

6.1 Introduction

In order to guide the reader for a better use of the indicators, the difference between

monitoring and evaluation needs to be clarified: Monitoring is the routine tracking of the key

elements of program/project performance through routine record keeping, routine regular reporting or continuous established surveillance systems. Monitoring helps managers to

take a decision in a timely manner. There is a need for consistency and harmonization in the

data being collected and indicators being used for measuring trends over time and comparisons between programmes, regions, or countries. Monitoring and evaluation are

related: monitoring contributes to evaluation and can inform the evaluator where to focus in

the in-depth evaluation. It is important to select a limited number of monitoring indicators that

will be actually used by implementers and managers for decision making. There is a tendency to collect to many data which are not used at all and do not provide any added

value for decision-making. This tendency should be avoided as it is time and money

consuming and put unnecessary burden on scarce human resources. Evaluation is the episodic assessment of the changes in targeted results related to the programme. It is more

difficult and more time and money consuming than monitoring due its methodological rigor

required to avoid wrong conclusions. Evaluation attempts to relate programme outputs to outcomes in behaviors or disease impacts after a certain period has passed. Even if the

programme is part of a collective effort, this kind of evaluation is very important. It helps the

managers to determine the added value of the programme investment.

7

Monitoring and evaluation take place at regular intervals: the interval is shorter for monitoring and longer for evaluation e.g. monthly, quarterly, annually, etc. Results are used for

reporting from peripheral operational levels to central level or from central level to donors.

Given the confusion met in categorizing indicators under input-process-output-outcome-

impact framework, we will not use these terminologies in the indicators described below. It was found however that some definitions are useful to assist those who want to use this

framework. Inputs refer to any resources (e.g. money, staff, time, vehicle, equipment and

other resources) required to achieve results; processes are activities (e.g. training, supervision, etc.) implemented by the programme using these resources; outputs are direct

results of the programme(e.g. number of staff trained, number of supervision visits, number

of patients on treatment, and any other coverage measure); outcomes are direct positive effects of the programme such as change in behavior (condom use, appropriate prescribing,

increased performance of trained staff, increased productivity, etc.) which ultimately would

lead to long term disease impacts as reduction in HIV incidence, reduction in opportunistic

infections, improved quality of life and reduction in mortality.

Below you will find a description of each component of medicine supply cycle and the key

aspects that could be monitored or evaluated. The most important aspects are covered by 11 core indicators. If potential problems in a particular PSM component are identified

through the core indicators, supplementary indicators can be used to get more information

about the problem and corrective measures can be implemented. Both lists of core and supplementary indicators are found in Annexes 1 & 2 respectively. The present document is

ready for field testing. After the field testing, the current indicators listed below and in

annexes 1 & 2 will be revised and finalized for publication.

6.2 Components of the Medicine Supply Cycle and related M&E Indicators

a. Selection

Treatment recommendations are updated regularly given the development of new more

effective and safer medicines, the availability of more generic ARVs on the market, the

occurrence of resistance and severe side-effects. The World Health Organization Department of HIV/AIDS updates the antiretroviral treatment guidelines every two years. It is

important that national HIV/AIDS programmes refer to updated WHO ARV treatment

guidelines when selecting medicines for their ART in order to guarantee the best possible ARV treatment. In addition, the efficiency of procurement is improved if programmes only

buy a limited number of nationally recommended medicines in sufficient quantities.

Core indicator 1: "Percentage of procured items that are on the national ARV

Standard Treatment Guidelines (STG)", measures the extent to which the ARV

procurement is in line with national standard ARV treatment guidelines (STG).

• Tools: The list of ARVs recommended in the national STG and the list of ARVs procured.

• Source of information: The information is collected at central level where the

procurement of ARV is done: the list of ARVs procured is compared to the list of ARVs recommended in the STG.

• Measurement: (Number of procured items in line with the national STG / Total

number of procured items)*100 • The target is 100%.

Although this indicator is easy to measure, most countries may find it not useful as no ARV

is procured outside the national STG. However, where discrepancy between procured ARVs and STG exists, this indicator is still relevant. This indicator can also be measured at facility

8

level during supervision visits or surveys in countries where some facilities receive ARVs

from several agencies outside the central medical stores.

If it turns out that procurement is not consistent with the national STG, supplementary issues

could be monitored such as the availability of updated national ARV STGs at central medical

stores and all health facilities to ensure that staff order the right products. Another issue is the regular revision of the national STG and the number of ARVs in the latest national ARV

STGs included in the national EML. The national STG should be regularly updated and the

national EML should be synchronized with the STG. Recommendation should be made to regularly update the national STG and to harmonize both documents (STG & EML) to

ensure that ARVs recommended in the STG are considered as essential medicines which

are tax free in several countries.

b. Prescription and use

Rational medicine prescription, selection and use are required to avoid the development of medicine resistance. Prescribing ART should be in line with national or WHO ARV Standard

Treatment Guidelines. Furthermore, irrational prescribing can distort the quantities of

medicines to be procured.

Core indicator 2 : "Percentage of patients receiving treatment regimens which are in

line with the ARV STG". Irrational prescribing behavior can affect procurement and supply management.

• Tools: requisition forms: when reordering ARVs at the central medical stores, health

facilities report on their requisition forms, the number of patients by treatment

regimen for quantification and procurement purposes. In some countries, the number of patients on the waiting list is also reported to show the unmet demand which is

real unlike the unmet need which is an estimate.

• Source of information: the information is collected at central level where the requisition forms are sent by health facilities.

• Measurement: (Number of patients with treatment regimens in line with the national

STG / Total number of patients on ARV treatment)*100. From this information, the

most used ARV treatment regimen can also be measured. • The target is 100%.

Another indicator which looks at rational medicine use is the "Percentage of ARV prescriptions that were dispensed in full at health facility level in the last 12 months" measures the extent to which ART prescriptions are dispensed in full. Normally, all

prescriptions should be dispensed in full. The targeted percentage for this indicator is

100%. This indicator is very difficult to measure as the required information for its measurement is not available and is difficult to collect. Reasons for not fulfilling a

prescription are numerous and include ARV shortage, poverty of patients unable to afford

the price of the whole prescription and irrational prescribing which would lead the pharmacist to modify the prescription by reducing the number of prescribed medicines if over-

prescription is detected. This indicator requires a survey to collect data and investigate the

reasons and disaggregate the results into groups according to reasons (ARV shortage,

poverty, irrational prescription).

If the prescription is not dispensed in full, treatment adherence is affected and resistance

may occur. Treatment adherence can also be monitored by measuring the proportion of patients who collect their refill according to the schedule and by the comparing the number

of pills remaining to the expected number of pills (use of pill counts of remaining ARVs when

patients return for refills). The adherence can be improved by informing patients on the

importance of respecting the schedule for ARV re-supply, correct dose and frequency of taking each ARV, and the use of appropriate containers for ARV dispensing.

9

c. Quantification and Forecasting

Interruptions of antiretroviral treatment should be avoided since this could lead to the development of resistance. Updated forecasts ensure that medicine quantification and

procurement activities are based on accurate data and the risk of stock outs is avoided or

minimized. The reverse, overstocking leading to expiry of products in the warehouse, should

also be prevented by keeping forecasts as close as possible to the needed quantities of medicines. Otherwise funds will be wasted on medicines that cannot be used. During recent

M&E workshops, participants mentioned that health facilities request quantities of medicines

based on the real demand (quantification). Quantification is a calculation of ARV quantities based on the real number of patients who need ARVs (demand). Forecasting is an estimate

of needed quantities of ARVs performed at national level based on various assumptions,

such as the estimated needs, the capacity of the health system and the available financial resources.

Core indicator 3: "Ratio between forecasted ARV quantities and procured ARV

quantities" r measures compliance to needed ARV quantities. The ARV quantities procured may be lower or higher than the forecasted quantities. As a target, the total quantities

procured should be as close as possible to quantities forecasted unless there is evidence

that forecast was not accurate or the budget to order the forecasted quantities and the needs have changed since the last forecast exercise.

• Tools: quantification/forecast report; annual quantities procured

• Source of information: the information is collected at central level where forecasting and procurement are carried out.

• Measurement: quantities procured for each ARV/quantities forecasted for each ARV

• The target ratio is as close as possible to 1

A ratio which is >1 means that quantities procured were higher than needs. Assuming that

procured quantities were consistent with forecasted quantities, it is useful to assess if the

consumption were consistent with the quantities procured.

Core indicator 4: " Ratio between consumed quantities and procured quantities"

measures overestimation of the actual ARV needs, which leads to expired ARVs in several

countries. Health facilities order quantities in line with the number of patients who need ARV treatment. If the quantities procured by the central medical stores were overestimated,

consumption will be very far below the quantities purchased. Overstock occurs with high risk

of expired medicines. • Tools: annual consumption of ARVs quantities; annual quantities procured

• Source of information: the information is collected at central level where annual

consumed ARV quantities and procured ARV quantities are registered. • Measurement: quantities procured for each ARV / quantities consumed for each ARV

• The target ratio is as close as possible to 1

A ratio which is >1 means that quantities procured were higher than the demand; the next forecasting exercise should be adjusted accordingly.

.

d. Procurement and Ordering: Prices and delivery time: Key issues in procurement of pharmaceuticals are product pricing and supplier delivery

times. The prices obtained for ARVs and other commodities have a significant influence on

the efficiency of the PSM system. In turn, the choice of a procurement method e.g. competitive bidding or direct procurement has an effect on the obtained prices.

Secondly, ensuring that ARVs are always in stock is important but at what cost? An efficient

programme manages to get the best prices and reliable deliveries from suppliers.

10

Core indicator 5: "Ratio between the median price paid by the country for each ARV to

the median international price" compares the obtained prices to those paid on the world market and measures the efficiency of ARVs procurement practices.

• Tools: median GPRM price for each ARV; median price of each ARV based on the

prices obtained from several orders during the year

• Source of information: the information is collected at central level where procurement is carried out.

• Measurement: national median price for each ARV/ Median GPRM price for each

ARV • The target ratio is <or= 1

If the ration is > 1, this means that the national median price is much higher than the median

international price (e.g. ratio=2 meaning that the country is paying twice the international

price) for a particular ARV medicine (single or Fixed Dose Combination). The causes should

be investigated e.g. were several quotations sought for a product? was the international competitive bidding used for multiple source ARVs as the main procurement method? or

were there a lot of emergency orders for which price negotiation is not an option? The value

and number of emergency orders have been included in the supplementary indicator list. In any case, corrective measures should be undertaken.

The number of emergency orders can also be used as a supplementary indicator (Supplementary indicator No. 8) for poor forecasting provided that the procurement was

done according to quantities forecasted i.e. bad forecasting (underestimation) at national

level leads to emergency orders. Other causes of emergency orders include health facilities

ordering lower quantities than quantities forecasted, delayed deliveries, etc.

Core indicator 6: "Percentage of non-emergency (regular) orders delivered in full and on time

as stated in the procurement agreement in the last 12 months for each supplier" measures supplier's performance in term of complying with the agreed delivery time and delivery of all

quantities ordered.

• Tools: list of regular orders from each supplier during the year

• Source of information: the information is collected at central level where procurement is carried out.

• Measurement: (number of orders from each supplier delivered on time and in full/total

number of orders from the same supplier)*100 • The target is 100% of orders should be delivered in full and within the stated delivery

time. This indicator is easy to measure and should be applied to each supplier at

central level. If delivery times are longer than the agreed delivery times, or if the delivered quantities do not match the ordered quantities, this may lead to ARV

shortage or to emergency order. Penalties for not complying with the agreements

(lower quantities, late deliveries) should be included in the contractual agreement

with the supplier. The delivery times can also be applied at district and facility levels if appropriate. In that case, this indicator measure the performance of the distribution

system as mentioned in core indicator 12. The reasons for late deliveries need to be

identified in order to attempt solving the problem.

e. Quality Assurance

To ensure that the procured products are safe and effective, certain quality standards should

be set to determine eligibility of manufacturers/suppliers. This could include Good

Manufacturing Practice (GMP) certification, WHO prequalification, or FDA, EU, stringent DRA approval, registration by the national regulatory authority, correct packaging, label

information and remaining shelf life upon receipt, and other quality standards defined in the

11

quality assurance policies. National drug regulatory authorities (DRA) and Organizations will

need to set their own quality standards and filter out the products that do not meet them.

Core indicator 7: "Percentage of ARV products which meet quality standards (or QC tests) ",

measures the extent to which ARVs procured meet the quality requirements. During a recent

IDA Solutions M&E course, several countries reported that QC tests are performed on each ARV batch before release for consumption. Therefore, this indicator on quality control is

easy to measure in countries where QC tests are performed on a regular basis before ARVs

are released for consumption, otherwise quality standards such as WHO prequalification or USFDA/EU/stringent DRA approval will be applied.

• Tools: ARV products meeting the defined quality standards; list of

prequalified/USFDA/Stringent DRA approved ARV products • Source of information: the information is collected at central level where quality

standards are assessed by the national DRA.

• Measurement: (number of ARV products which meet the quality standards/total

number of procured ARV products)*100 • The target 100% products meet the quality standards. Products that failed the quality

standards in particular those failing the QC tests need to be sent back to the

supplier/manufacturer or destroyed on site and replaced at the supplier's cost. This should be reflected in the contractual agreement with the supplier. Quality control is

done in addition to other nationally or/and internationally defined quality standards

used to assess the extent to which procured products are of good quality.

In addition to the QC test any observation made on product quality during inspection

activities should be reported (Supplementary indicator No. 12). This could for instance

pertain to product packaging and labeling or any other abnormalities observed during inspection activities of the pharmaceutical products delivered. A supplementary indicator on

this topic has been included in Annex 2. If the remaining shelf life of received products is

below the set standard, this can be measured by using another supplementary indicator in the QA section (Supplementary indicator No. 13). If there are problems with reporting

product quality issues and it is unknown whether facilities are using a systematic approach,

this could be measured by using a qualitative supplementary indicator i.e. percentage of

facilities that have procedures in place to report product quality issues (Supplementary indicator No. 14).

f. Inventory control (manual or computerized) Inventory control for stock management is a useful management tool and an important

source of information for monitoring the PSM system. It is important that inventory control

reports are sent from the different levels to central level in a timely fashion to facilitate data analysis, decision-making and reporting.

Core indicator 8: "Percentage of health facilities that submit their inventory control

reports according to an established schedule." measures regularity in reporting. • Tools: list of health facilities having submitted their inventory control reports

• Source of information: the information is collected at central level where health

facilities submit their inventory control reports. • Measurement: number of health facilities which have submitted their inventory control

reports/ total number of health facilities expected to submit their reports.

• The expected target is 100% of facilities submitting their reports. This indicator is easy to measure as information is available and is useful for effective M&E system.

It is also important that data captured in the system is consistent with the actual situation in

the warehouse. "Percentage of products for which the physical counts and inventory records (stock cards or computerized data) match" measures the accuracy of inventory

control and the quality of reported data i.e. whether inventory control data and actual

12

physical stock data match. This indicator was not considered as a core monitoring indicator

as it is only available in countries where supervisory visits are done on a regular basis and systematically. The indicator is measured using supervisory report information as the

information is collected through physical counts carried out during supervision visits unless

on site facility survey is conducted. The expected target is 100%.

Core indicator 9: "Percentage of health facilities with all ARV orders made while the

ARV stock on hand was within the security stock level during in the last 12 months." measures the effective use of inventory control.

• Tools: requisition forms from health facilities.

• Source of information: the information is collected at central level where health facilities submit their requisition forms.

• Measurement: (number of health facilities which had all ARV orders made while the

ARV stock level was within the security stock level for each ARV / total number of

health facilities submitting requisition forms)*100. • The expected target is 100% of health facilities with orders done within the

established safe inventory control levels. If the ARV order is done when the stock

level of an item is below the established security stock level, the risk of shortage is high. This indicator shows whether inventory control is being used properly or if

product requests are sent to the procurement department too late, which can be a

possible cause for stock outs.

g. Availability of ARV medicines

Interrupted ART can lead to treatment resistance. Therefore product availability is a key factor for any PSM system and must be ensured at all levels of the health system.

Core indicator 10: "Percentage of health facilities dispensing ARVs that had no stock out of one or more ARVs in the last 12 months " measures the performance of the PSM

system in term of ARV availability. This indicator is easy to monitor as this information is

provided regularly to the central medical stores in order to receive new ARV supplies. It is

useful indicator for assessing the effectiveness of a PSM system. ARVs should be always available at the health facility level to ensure that all prescriptions are filled when patients

need ARVs.

• Tools: ARV stock inventory control reports from health facilities; ARV requisition forms.


facilities submit their inventory control reports or their ARV requisition forms. • Measurement: (number of health facilities which had no ARV stock out / total number

of studied health facilities dispensing ARVs)*100.

• The expected target is 100% of health facilities with no ARV stock out or 0% facility

experiencing ARV stock out. A related supplementary indicator is the duration of stock outs since this will show the severity of the stock out problem. The longer the

duration of the stock out, the more severe is the situation (Supplementary indicator

No. 15).

h. Distribution

Timely distribution is important since it is related to product availability and the need for uninterrupted ART supply at health facility level.

Core indicator 11: "Percentage of facilities which received all ARV orders in full in the

last 12 months measures the reliability of distribution system. • Tools: requisition forms from health facilities.

13


facilities submit their requisition forms. • Measurement: (number of health facilities which received their ARV orders in full in

the last 12 months / total number of health facilities dispensing ARVs)*100.

• The expected target is 100% of facilities. This indicator is similar to core indicator 6

discussed above but measures two different issues: indicator 6 measures the performance of the suppliers while indicator 10 measures the performance of the

distribution system.

A deeper analysis of which facilities received all deliveries on time, and which ones did not

receive them on time can give more information on causes: for instance 10 facilities are

expecting 10 deliveries each. 5 facilities received all deliveries in full and on time (100% deliveries received in full and on time for 5 sites) and 5 facilities had only 8 deliveries in full

and on times (80% deliveries on time). The characteristics of these facilities for which

deliveries did not arrive on time could be useful for instance treatment sites in remote areas

with bad roads and during rainy season could explain this problem. In the other words, late deliveries could be internal e.g. the order was received too late compared to the standard

delivery schedule or external e.g. bad roads or heavy rains.

Finally, appropriate storage conditions e.g. warehouse temperature, cool storage for certain diagnostics, first expiry first out policy, and secured storage for ARVs need to be maintained.

This can be measured using the supplementary indicator on acceptable storage conditions,

which need to be defined beforehand (Supplementary indicator No. 17).

The indicators mentioned in the following sections are all supplementary indicators and are

found in annex 2.

i. Human Resources

An important issue in HIV/AIDS-related human resources is the presence of sufficient and

appropriately qualified staff to enroll, treat and counsel patients on ART. Human resource shortage is a significant bottleneck of ART scale up in most developing countries. The first

supplementary indicator of this section measures the percentage of facilities that exceed the

recommended standard of ART patients per dedicated person to dispense ART prescriptions

(Supplementary indicator No. 19).

Another important aspect of human resource is the number of qualified staff allocated for

M&E, completing various reports. When the number of staff allocated to these activities is inadequate, the burden becomes high and the quality of reported data and reporting rates or

timely reporting becomes low.

A second important issue in HIV/AIDS-related human resources is the level of staff training.

Staff should be trained in various areas related to their responsibilities. Pre-service and in-

service training in ART PSM is essential to staff. Refresher courses should be provided

regularly e.g. once a year to maintain training impact and to provide updated information. The second supplementary indicator of this section measures the percentage of staff

responsible for HIV/AIDS services trained in HIV related commodity management


14

j. Funding Sources and Budgeting

Funding for ART can originate from national and international sources and should be used in efficient and transparent manner. Developing a national procurement plan for HIV/AIDS that

has been approved by all partners is a first step in ensuring efficient use of financial

resources and also in preventing the duplication of efforts. This plan should contain the

budget, the quantities of products to be procured and the sources of funding among other things. Secondly, a mechanism for annual audits should be set up to ensure that funds are

used appropriately and transparently. These aspects can be measured using the first two

supplementary indicators of this section: the existence of a national procurement plan (Supplementary indicator No.21) and the existence of an annual audit of a Central Medical

Store (Supplementary indicator No.24).

Other issues that can be monitored include percentage of late release of funds from ministry

of finances to procurement services (e.g. Central medical stores) and percentage of late

payments to suppliers (Supplementary indicator No.22). Late release of funds by the

government or partner may jeopardize the supply system.

Late payments can result in penalties from the suppliers and the cause should be

investigated after which preventive action can be taken. Finally, the level of operational costs can be monitored as an indicator of efficiency. For this supplementary indicator the total

commodity value needs to be divided by the total cost of warehousing and distributing the

commodities (Supplementary indicator No. 23). As with other quantitative indicators a target needs to be set so that an investigation is started if the measurements are below this target

value.

k. Organization and Management Written standard operating procedures (SOP) and guidelines for the activities of the PSM

cycle are required to ensure quality assurance throughout the PSM system. If standard

procedures are followed there is little room for errors due to personal interpretation of the required actions and their frequency. Supervision will be required to make sure that SOPs

are actually adhered to. Finally, guidelines to ensure that supervision is executed

appropriately are also required. Supervisory visits should be conducted in a standardized

way rather than them being entirely dependent on the person who undertakes them. Therefore the three supplementary indicators in this section pertain to the existence of

written management procedures and tools (Supplementary indicator No. 25), the number of

supervisory visits made compared to the number of scheduled visits (Supplementary indicator No. 26) and the existence of guidelines for supervisory visits (Supplementary

indicator No. 27).

l. Policies, Legislation and Regulations

The policy, legal and regulatory environment is meant to ensure safety of pharmaceutical

products and equitable access to health services and treatment. Compulsory registration of

pharmaceuticals is a way a government ensures that the products used in the country meet certain predefined standards. However, registration can take several months to several

years, depending on the country. In view of the HIV/AIDS treatment urgency in many

developing countries, a fast track registration system for ARVs should be set up. The existence of such a system is one of the supplementary indicators of this section


Since the provision of ART services requires specific standards including human resources

with specific knowledge, skills and capacity for effective ART PSM, it is important that a

facility starting the provision of ART can refer to nationally agreed quality standards.

Therefore, supplementary indicator No.29 measures the existence of criteria for ART preparedness or a national accreditation programme so that the minimum requirements for

quality ART services are guaranteed.

15

Another supplementary indicator in this section is related to financial barriers that prevent equitable access to health services and treatment. If these barriers exist the government can

opt to set up a policy on user fees and exemptions so that no one is excluded from HIV/AIDS

services and treatment in the public health sector. The related indicator which measures the

existence of such a policy is Supplementary indicator No. 30.

Detailed information on the core indicators and on the supplementary indicators are found in

Annex 1 and in Annex 2 below.

16

7. Annexes

Annex 1: Core Indicators for Monitoring and Evaluation of the Performance of the ARV Procurement & Supply Management Systems

No. PSM stage Core Indicator Purpose/

When to use How to measure Target &

LEVEL Frequency Linked

Supplementary Indicators

Required information And Data source

1. Product selection

Percentage of items procured that are on the national ARV Standard Treatment Guidelines (STG).

To measure whether ARV procurement is in line with national ARV STG

Numerator: No. of items procured found in ART STG Denominator: Total No. of items procured

100% at central level

As often as necessary for intermediate reporting)

Annex 2, # 1, 2.

Required Info: National ARV treatment guidelines, EML, procured ARV list. Data source: Ministry of Health, National AIDS Committee, CMS or other national ARVs procurement agency if different from the CMS

2. Prescribing & Use

% patients receiving treatment

regimens in line with the ARV

STG.

To measure whether ARV prescriptions are in line with national ARV treatment guidelines

Numerator: No. of patients with ARV regimens in line with national ARV STG Denominator: Total No. of patients on ART

100% at Facility level

Annually (and as often as necessary for intermediate reporting)

None Required Info: National ARV treatment guidelines, ART prescriptions. Data source: Health facility treatment records, MOH

3. 4.

Forecasting Ratio between forecast ARV quantities and ordered ARV quantities for each ARV.

May be useful to measure accuracy of forecasting in countries where expired ARVs due to overestimation is an issue. Importance of expired ARVs measured by supplementary indicator No.16 and the discrepancy between consumption and supplies can be measured as follows:

Percentage variation between

(1) procured (or supplied in

To measure the extent to which the quantities of ARVs ordered are consistent with the estimated needs (quantities of ARVs forecasted). To measure the extent to which

Numerator:

quantities procured for each ARV

Denominator:

quantities forecasted for each

ARV

Numerator:

quantities procured

for each ARV

Ratio = 1 at central level


Annex 2, # 7 Required Info: Ordered ARV quantities and forecast ARV quantities. Data source: MOH, National AIDS Programme, CMS and other places where can be found ARV Forecasts and procurement records

18

No. PSM stage Core Indicator Purpose/ When to use

How to measure Target & LEVEL

Frequency Linked Supplementary Indicators


case of push system) quantities (2) consumed quantities

the quantities of ARVs procured match the demand (quantities of ARVs correspond to the number of patients in the health facilities).

Denominator:

quantities consumed for each

ARV

Ratio = 1 at central level Ratio>1 means overestimation or oversupply.

5. Procurement & Ordering

Ratio between the median price

paid by the country for a specific ARV to the median international

price (medicine by medicine

basis).

To measure the efficiency of the procurement practices: the national median price is compared to the median international price.

Numerator: Median price paid for an item over several orders. Denominator: Median international price for same item. GPRM median price for a given ARV medicine is the reference standard.

Ratio<or=1 meaning lower or at most equal to the median international price (e.g. GPRM, MSH) Central level


Annex 2, # 8, 9

Required Info: Prices of ARVs ordered ; procurement records; international Median ARV prices from the GPRM data base. Data source: MOH, National AIDS Programme, CMS and other places with ARV order records, WHO GPRM database.

6 Procurement & Ordering

Percentage of non-emergency (regular) orders delivered in full and on time as stated in the procurement agreement in the last 12 months for each supplier

To measure supplier's performance in terms of complying with the agreed delivery time. The delivery times should be less or equal to the stated delivery times.

Numerator: number of non-emergency orders delivered in full and on time according to the agreement in the last 12 months for each supplier. Denominator: Total number of non-emergency orders from each supplier in the last 12 months.

100% orders Central level, District level and Facility level


None Required Info: procurement records; warehouse receipt notes. Data source: MOH, National AIDS Programme, CMS and other places with ARV order receipt notes.

19





.7 Quality Assurance

Percentage of products tested that met the quality control tests.

To measure product quality.

Numerator: Number of products that met quality tests performed by a WHO prequalified QC lab. Denominator: Total number of products tested by a WHO prequalified QC lab.

100% products meet the quality standards or 0% failure. Central level

Each supply none Required Info: national pharmaceutical standards; QC reports. Data source: QC Labs, CMS

8. Inventory control

Percentage of facilities submitting complete inventory control reports according to an established schedule.

To measure regularity reporting.

Numerator: Number of facilities submitting complete inventory control reports according to schedule. Denominator: Total number of facilities required to submit reports.

100% Central, regional, provincial, district levels (all levels receiving reports)


none Required Info: Number of facilities submitting complete reports on time, total number of facilities in reporting area, reporting deadlines. Data source: MOH or CMS reports on reporting schedule, inventory control reports and their submission dates.

9. Inventory control

Percentage of health facilities with all ARV orders done while

the ARV stock level on hand is

within the security stock level

during in the last 12 months

To measure effective use of inventory control.

Numerator: number of health facilities

which had all ARV

orders made while

the ARV stock in hand was within

the security stock

level for each ARV

Denominator: total

number of health

facilities submitting requisition forms

100% Central, district levels


Annex 2, # 19 and number of emergency orders in a year.

Required Info: Delivery schedule Data source: LMIS, ARV receipt reports.

20





10. Availability

Percentage of health facilities dispensing ARVs that had no stock out of one or more ARVs in the last 12 months

To measure the performance of the PSM system in term of ARV availability.

Numerator: No of facilities dispensing ARVs that had no stock out of one or more ARVs in the last 12 months Denominator: Total No of facilities dispensing ARVs surveyed.

100% health facilities without stock out or 0% facilities with stock outs Central, district levels


Annex 2, # 16, 17, 18, 19

Required Info: Stock levels Data source: LMIS, stock cards and other stock management tools.

11. Distribution Percentage of facilities which

received all ARV orders in full in

the last 12 months .

To measure reliability of distribution system: all health facilities should receive all deliveries on time to avoid ARV stock out.

Numerator: number of facilities that received all ARVs orders in full within the last 12 months Denominator: Total number of facilities

100% Central, district levels


None Required Info: Minimum stock levels per product, stock levels at time of each supply was received, total number of products re-supplied during the last 12 months. Data source: stock cards and other stock management tools.

Annex 2: Supplementary Indicators for M&E of ARV Procurement & Supply Management Systems

No. Level in

the supply chain

Supplementary Indicator Definition Frequency Required information

Data source

Product Selection

1 Facility Latest copy of national STGs available and seen at dispensary at ART site.

Latest copy of national STGs (or facility formulary, if relevant) reported to be available and seen at health facility


STGs, EML, product list.

Observation at health facilities.

2 Central Percentage of listed ARV medicines in latest national STGs included in latest national EML

Numerator: Number of listed ARV products in latest national ARV treatment guidelines included in latest national EML Denominator: Total number of ARV products included in latest national guidelines


STGs, EML, product list.

National selection committee, Ministry of Health

3 Central National ARV resistance surveillance system exists and has reported results to national selection committee in last 12 months

National ARV resistance surveillance system exists and has reported results to national selection committee in last 12 months


Components of a functioning ARV resistance surveillance system.

National selection committee, Ministry of Health

Prescription and Use

4 Facility Percentage of patients overdue in collecting prescriptions who were identified and appropriate action taken for follow-up promptly This indicator measures patient's treatment adherence and corrective action taken. It is an interesting indicator but very cumbersome. A less

Numerator: Number of patients overdue collecting prescription who were identified and followed up promptly as per procedure. Denominator: Number of patients overdue in collecting ART prescriptions


Definition of when a patient is classified as overdue. Procedure for action to be taken at pharmacy if patient is overdue in collecting ARV prescription, including timeline.

Patient-focused records, diary, dispensing records.

22

No. Level in the supply chain


Data source

cumbersome way of looking at this issue but gives different information: "The proportion of facilities that track overdue patients routinely or the proportion of facilities reporting overdue patients"

Numerator: Number of facilities tracking overdue patients or reporting overdue patients. . Denominator: Number of facilities surveyed.

5 Facility Percentage of encounters where pharmacy staff members explained the dose and frequency of each ARV medicine to the patient

Numerator: Number of encounters where pharmacy staff members explained the dose and frequency of each ARV medicine to the patient/caregiver Denominator: Total number of encounters where pharmacy staff members issued ARVs to patient/caregiver


Prescribed dose and frequency

Data collection tools, observation of pharmacy staff members; supervisory check list

6 Facility Percentage of prescribed ARVs dispensed in an appropriate container with clearly labeled identity, with clearly labeled dosage instructions

Numerator: Number of prescribed ARVs that are dispensed in appropriate and clearly labeled containers Denominator: Total number of prescribed ARVs dispensed


Standards for appropriate containers; standards for clear labeling

Data collection tools, observation of dispensed ARVs

Forecasting

23



Data source

7 Central Percentage of forecasts and updates completed as per standard operating procedures

Numerator: Number of forecasting exercises completed in a time period Denominator: Number of forecasting exercises that were scheduled to be conducted in the time period


Forecasts Forecast records

Procurement

8 Central Percent of emergency orders issued in the last 12 months (emergency order is any unplanned order done to solve an urgent imminent shortage.

Numerator: Number of emergency orders Denominator: Number of total orders


Number of purchase orders issued; number of emergency orders issued

Transaction records

9 Central Proportion of the value of emergency orders issued in the last 12 months

Numerator: Total value of emergency orders (in money and quantities). Denominator: Value (in money and quantities) of total orders.


Total value of purchase orders issued; total value of emergency orders issued

Transaction records

10 Central Proportion of orders which were fully filled in the last 12 months

Numerator: number of orders which were fully filled in the last 12 months (orders where quantities received = at least quantities ordered) Denominator: total number of orders in the same period


Requisitions and issues at all levels (i.e., quantities of products ordered and quantities received) for a specific time period

Requisition and issue records; LMIS reports

24



Data source

11 Central Proportion of ARV products found in the stores which are listed in the latest national ART STGs

Numerator: Number of ARV products found in the stores which are listed in the latest STGs Denominator: Total number of ARV products found in the stores.


List of products in STGs

Inventory records

Quality Assurance

12 Central General product quality issues Any observation made on product quality outside of the QC test.

Each supply Product quality observations

Products upon arrival. contract with supplier

13 Central Percentage of items received where the minimum shelf life equaled or exceeded the shelf life specified on the purchase order.

Numerator: Number of items received where the minimum shelf life equaled or exceeded the shelf life specified on the purchase order. Denominator: Total number of items received.


Shipment invoices; shelf-life regulations per country

Purchase orders; transaction records

14 Central Percentage of health facilities that have procedures in place to report product quality issues.

Numerator: Number of health facilities that have procedures in place to report product quality issues. Denominator: Total number of health facilities.


System available to raise issues

25



Data source

Warehousing and Inventory Control

15 Central, District, Facility

Stock out duration: average percentage of time out of stock for ARV products during a specified time period.

Numerator: Sum of all days out of stock for each ARV product during a specified time period (e.g. 180 days or 365 days). Denominator: Total number days for the specified time period x total number of items Or Numerator: sum all days of stock out and divide by the number of items (=average stock out days) ex. (10+20+30)/3=20 Denominator: Total number days for the specified time period (ex. 180 days)


Designated stock levels, stock out data, review of stock cards, consumption data.

LMIS, stock cards, consumption records,


Product losses due to expired drugs, damage, theft and other diversion: Value lost per value received

Numerator: Value (quantities and financial) of product lost due to: (1) expired products (2) damage, (3) theft and other types of diversion. Denominator: Total value (quantities and financial) of product received.


Value of product lost due to theft, damage, diversion; total value of products received

Stock cards, transaction records, inventory records, any other stock management records

26



Data source


Percentage of storage facilities meeting acceptable storage conditions and handling procedures

Numerator: Number of storage facilities meeting acceptable storage conditions and handling procedures. Denominator: Number of facilities visited.


Checklist of acceptable storage conditions, SOPs.

Checklist of acceptable storage conditions, SOPs.

Distribution


Emergency order lead time Average duration of the time between date emergency order placed and when received.


Dates emergency orders were placed and received.

LMIS, goods receipt notes.

Human Resources

27



Data source

19 Facility Percentage of facilities that meet or exceed the recommended standard of ART patients per dedicated staff available to dispense ART prescriptions.

Numerator: Number of facilities which meet or exceed the recommended standard of ART patients per dedicated staff Denominator: Number of facilities surveyed


Number of staff, number of ART patients, national standard of ratio of staff/ART patients (ex. 10 qualified staff/1000 ART patients).

Dispensing records, staff schedule.

20 Facility Percentage of staff responsible of PSM of ARVs who have been trained in PSM

Numerator: Number of staff responsible of ARV PSM who have been trained in ARV procurement and supply management. Denominator: Total number of staff responsible of ARV PSM


Number of staff responsible of PSM of ARVs and other HIV-related commodities who have been trained in ARV/HIV related commodities procurement and supply management.; Number of staff responsible of PSM of ARVs and other HIV-related commodities

Ministry of Health, training officer, national training center, training reports

Funding Sources and Budgeting

21 Central A national consolidated procurement plan exists and has been approved/funded by all partners

A national consolidated procurement plan exists and has been approved/funded by all partners.


Fund allocation records; accounting books and records

Ministry of Health

22 Central Percentage of late payments to suppliers by the central medical stores

Numerator: Number of payments made late to suppliers. Denominator: Total number of

Annually (and as often as necessary for intermediate

Procurement contract terms; invoices

Accounting records

28



Data source

payments to suppliers. reporting)

23 Central Proportion of total commodity value (costs in money) to total cost of the central medical stores in the last 12 months

Numerator: total commodity value (costs in money) in the last 12 months Denominator: Total costs of the medical stores in the same period.


Operational costs: warehousing, distribution, transport

Medical store finance department

24 Central Existence of an annual audit of the Central Medical Store.

Annual audit conducted and results available.


Daily financial statements; accounting books and records

Ministry of Health

Organization and Management

25 Facility District Central

Existence of appropriate organizational and management procedures and tools

Written standard operating procedures and guidelines (e.g., manuals, job aids, standards) exist to carry out logistics responsibilities such as using LMIS forms, forecasting/ordering commodities, inventory management, storage, distribution, product selection, budgeting, supervision and staff development, etc.


Written Standard Operating Procedures for specific logistics responsibilities

Observation of written procedures and guidelines (e.g., manuals, job aids, standard operation procedures) at central level; observation of existence at lower levels

29



Data source


Percent variation between the number of supervisory visits made and the number of supervisory visits scheduled

Numerator: total number of scheduled supervisory visits - total number of supervisory visits made. Denominator: total number of scheduled supervisory visits.


Records of supervisory visits. Established schedule of supervisory visits

Supervisory records


Existence of guidelines for appropriate supervision system to support the operationalization of logistics activities

Written guidelines exist for routine supervision to support logistics operations at facilities.


Guidelines for routine supervision system that covers logistics activities

Observation of guidelines for routine supervision at central level; observation of existence at lower levels

Policies, Legislation and Regulations

28 Central Existence of a system for fast-track registration of ARVs in countries where national registration is required for HIV/AIDS commodities.

A system for fast-track registration of ARVs exists.


Components of a functioning fast track registration system.

Ministry of Health, Regulatory authority

29 Central Existence of criteria for ART preparedness or of a pharmaceutical facility accreditation programme for ART at national level.

ART preparedness criteria or an accreditation programme exist at the national level for ART programmes.


Criteria for ART preparedness or for national accreditation programme

Ministry of Health

30 Central Existence of a policy on user fees and exemptions for HIV/AIDS treatment and services.

A national policy on user fees and exemptions for HIV/AIDS treatment and services exists.


Components of a policy on health system user fees and exemptions.

Ministry of Health

8. References

Presentations made in the meeting (available on

http://www.who.int/3by5/amds/geneva_oct10/en/index.html):

! 4 WHO presentations: o Indicators for monitoring ARV treatment outcomes (URL)

o George Loth: Patient monitoring (URL)

o Marco Vitoria: Pharmacovigilance (URL) o Cyril Pervilhac: HIV Drug Resistance (URL)

! Xenophon Santas, U.S. President’s Emergency Plan for AIDS Relief Reporting

Requirements and the Country Operational Plan and Reporting System (COPRS)

! Maponga CC et al. Presentation on "Desk Study Conducted in Preparation for

Discussion on Harmonization of National Reporting Requirements for ART Medicines Supply Management". IDA Solutions, Desk Study for AIDS Medicines and Diagnostic

Services, October 6, 2005

• Yasmin Chandani. Presentation on "Reporting Requirement and Indicator

Harmonization for HIV Medicines and Diagnostics: a Review of PEPFAR, GFATM

and Country Specific Requirements and Indicators". Prepared by JSI for AIDS

Medicines and Diagnostics Services, JSI 28 September 2005

! Douglas Keene, Overview of Ethiopia ARVs and other Medicines Supply

Management Issues

! Ernest Rwagasana, Donor reporting requirements for ARV’s and related drugs in,

Rwanda

! Emma Nelson Msuya , Harmonization of Reporting Requirements: Tanzania

! Martin Olowo Oteba , Harmonization of donor reporting requirements for antiretrovirals and related drugs: Uganda (URL)

! Nguyen Van Kinh , Accessing to ARV in HIV/AIDS care and treatment programme in Vietnam (URL)

Reference documents:

1. Maponga CC et al. Desk Study Conducted in Preparation for Discussion on

Harmonization of National Reporting Requirements for ART Medicines Supply

Management. IDA Solutions, Desk Study for AIDS Medicines and Diagnostic

Services, October 6, 2005 http://www.who.int/3by5/amds/geneva_oct10/en/index.html

2. JSI. Reporting Requirement and Indicator Harmonization for HIV Medicines and Diagnostics: a Review of PEPFAR, GFATM and Country Specific Requirements and

Indicators. Prepared by JSI for AIDS Medicines and Diagnostics Services, JSI 28

September 2005. http://www.who.int/3by5/amds/geneva_oct10/en/index.html

3. WHO Indicators for monitoring national drug policies: a practical manual. Second

edition, WHO/EDM 1998

31

4. WHO. HIV Patient ART Monitoring Meeting. Report of 29-31March 2004 meeting at ICC Geneva http://www.who.int/hiv/strategic/me/artmonitoring/en/index.html

5. The Global Fund to Fight AIDS, Tuberculosis and Malaria: Monitoring and Evaluation Toolkit: HIV/AIDS, Tuberculosis and Malaria. Annexes: Selected indicators for

HIV/AIDS, Tuberculosis and Malaria. June 2004

http://www.theglobalfund.org/en/links_resources/library/evaluation_framework/

6. The President’s Emergency Plan for AIDS Relief: Indicators, Reporting

Requirements, and Guidelines for Focus Countries. Revised for FY2006 Reporting. July 29, 2005.

7. The WHO M&E Webpage, a good concentration of already done M&E work: http://www.who.int/hiv/pub/me/en/

8. World Health Organization. Monitoring and Evaluation Toolkit HIV/AIDS, tuberculosis

and malaria June 2004 http://www.who.int/hiv/pub/me/me_toolkit2004/en/print.html

9. WHO, 2005: NATIONAL AIDS PROGRAMMES A guide to indicators for monitoring

and evaluating national antiretroviral programmes http://www.who.int/hiv/pub/me/naparv/en/index.html

Documents

Harmonization of Monitoring and Evaluation Indicators for ARV … · Harmonization of Monitoring and Evaluation Indicators for ARV Procurement and Supply Management Systems Working