Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Center for Interdisciplinary Addiction Research
Evaluating the Joint Action HA-REACT
Evaluation framework, methods and tools
Center for Interdisciplinary Addiction Research
Challenges of evaluating the JA
Timeframe: October 2015 – September 2018
Overall objectives of the project Zero new HIV cases, reduced HCV and TB among PWID in the EU by 2020
Improved prevention and treatment of blood-borne infections and TB in priority regions and priority groups in the European Union
AIM: To improve the capacity to respond to HIV and co-infection risks and provide harm reduction with specific focus on people who inject drugs (PWID) in priority regions in the EU
6 work packages with – different objectives
– activities either in community or in prison
– implemented in different EU focus countries (Latvia, Lithuania, Hungary, Poland and Czech Republic)
Center for Interdisciplinary Addiction Research
Basis for the evaluation
WP2. Dissemination
WP4. Testing and linkage to care
WP5. Scaling up harm reduction
WP6. Harm reduction and continuity of care in prisons
WP7. Integrated care
WP8 Sustainability and long-term funding
Logical framework approach (LFA)
Center for Interdisciplinary Addiction Research
Basis for the evaluation
WP2. Dissemination
WP4. Testing and linkage to care
WP5. Scaling up harm reduction
WP6. Harm reduction and continuity of care in prisons
WP7. Integrated care
WP8 Sustainability and long-term funding
Logical framework approach (LFA)
Output and outcome indicators according to Grant Agreement
Assessment of effects among professionals of Focus countries
and in EU Member States
Center for Interdisciplinary Addiction Research
Objectives of the Evaluation
Process evaluation
– Monitor if the project activities in the WPs have been implemented as intended
– Process indicators: accuracy of content, timeline, participants
– Performance indicators: participants satisfaction, increase in skills
Outcome evaluation
– Evaluate if the outputs (materials, guidelines, training manuals) have been achieved, distributed and known to a wider audience
– Indicators: 100% distribution to target group, knowledge to a wider audience in respective Focus Country
Effect evaluation
– Evaluate if overall project objectives and specific objectives of the WP have been achieved
– Indicators: Experts assessment of improvements due to HA-REACT activities
Center for Interdisciplinary Addiction Research
The evaluation framework
• Mission reports, participants lists
• Participants’ satisfaction rating
• Pre- and post-testing of trainings Processes
• Basically deliverables, products developed within the project
• Website, manuals, educational material Outcome
Impact • Effects of the project activities in Focus
Countries • Surveys among different expert groups in EU
Center for Interdisciplinary Addiction Research
Evaluation methods
Standardised questionnaire for
participant satisfaction in 7 languages
Analysis of HA-REACT web server logs
Support of questionnaires for HIV/HCV testing, for the mobile unit in LT and the condom provision in CZ
Materials produced
Yearly online survey among different stakeholders
Focus group interviews in Focus Countries
Individual interviews with policy maker in Focus Countries
Processes
Output
Impact
Center for Interdisciplinary Addiction Research
Selected results
Participant satisfaction
Quality of the HA-REACT products
Visibility of the HA-REACT project
Impact of the HA-REACT project in extended
Focus Countries (HU, LV, LT, PL, CZ)
Center for Interdisciplinary Addiction Research
Results – participants satisfaction
A total of 38 activities with a total of 835 participants were evaluated. The
participant satisfaction questionnaire was answered by 81% (n=676)
89% 90%
92% 86%
94% 83%
90% 98%
90% 91%
WP4: Trainings on HIV/HCV testing
WP 5: Debatte in Riga und Vilnius
WP 5: Study visit in Spanien, Prag, Helsinki
WP5: Workshops zu Harm Reduction, Opioid…
WP 6: Study visit in Luxemburg und Berlin
WP 6: Internationales Seminar in Warschau
WP6: Train the trainer in Popow
WP 7: Study visit nach Portugal
WP 7: Nationale workshops in Tallinn, Vilnius, Prag
WP 7: Internatioaler Workshop in Utrecht
Average satisfaction rate
Center for Interdisciplinary Addiction Research
Yearly Online Survey
Response rate (%)
Target Group 2016 2017 2018
Associated partners (n=21) 95 81 95
Advisory board (n=16) 77 50 23
THL (n=1) 100 100 100
Collaborating partners (n=9-14) 33 29 23
Heads of REITOX Network (n=27) 52 33 37
EHRN (Eurasian HR Network) (n=10-45) 20 40
Center for Interdisciplinary Addiction Research
Challenges of the evaluation
Problems in congruence
- Timeline of activities was partly unclear and not always specified in the LFA workshops
- Some discrepancy in indicators specifies in grant agreement documents and log frame workshop
Problems of cooperation with evaluation
- Utilisation of evaluation instrument was difficult in the beginning and for certain activities such as international workshops
- Partly the responsibility for evaluating events was unclear (WP lead or local event organisers)
Problems of output evaluation
– Requires to forward the products which have been developed
Problem of impact evaluation
– Effects often occur later and not within the life of the project
– Multiple influences in countries through changing policy makes it difficult to assess the project effects
Major requirements
Constant own monitoring and communication across all partners involved in activities
Emphasising again and again the importance of evaluation