A draft report of the Evidence Synthesis and Application for Policy and Practice project

ESAPP Review

A draft report of the Evidence Synthesis and Application for Policy and Practice project

May 2013Graham Brown, Kylie Johnston and Jeanne Ellard

Australian Research Centre in Sex, Health and Society

www.latrobe.edu.au/arcshsMelbourne, Australia

HIV response in Australia

• Australian HIV response has long recognised that interventions working across multiple social, political, economic, behavioural and health service conditions, operating within supportive environments, are more likely to affect behaviour than those interventions working at one level

• Resurging and emerging epidemics • Role of antiretroviral (ARV) treatments in preventing HIV transmission • Continued barriers at a structural level reducing or undermining impact• Recognising the need to better integrate biomedical, individual, community and

structural approaches for HIV prevention - coined “combination prevention”

Evidence gaps

• Shared evidence base is not consistent across strategies or has not been maintained as the contexts have continued to change.

• Understanding of what works - but less so why, or in what combination.

• This undermines the strength of programs and organisations, and the capacity to adapt to changing environments with confidence.

• Highlighted within Implementing the UN Declaration Report and Melbourne Declaration

identify the areas of HIV prevention where the published evidence of effectiveness and quality practice is most, modest, or least developed;

Section 2.0 of full report

identify the monitoring and evaluation methods used in day to day practice in community organisations to contribute to that evidence (Australia and similar epidemics);


review of capacity-building approaches in Australia and internationally to increase the quality of evidence being developed in community-based HIV health promotion;


develop a draft Monitoring, Evaluation and Learning framework for community HIV prevention to support building evidence for policy and practice


Develop an draft example application of the Monitoring, Evaluation and Learning Framework to community based HIV prevention and health promotion


Evidence most, modest and least developed

• Review of systematic reviews, economic reviews, narrative reviews and commentaries on the evidence to guide the prevention of sexual transmission of HIV in concentrated epidemics (2005+).

• Published evidence from research and practice (reduced to ~130 articles)

• Additional focus - three priority groups identified due to the likely impact of testing and treating approaches as well as experiencing resurging or emerging epidemics. – PLHIV -Gay Men -Priority culturally and linguistically diverse (CALD) communities

Generally Implementation evaluation and quality practice indicators with specific target groups

Example program Evidence on what does or does not work

Evidence on how it works (including how to adapt to context)

Gay men PLHIV Priority CALD communities in western countries

Health promotion Systems

how the interventions interact and impact together

Least developed Least developed Least developed Least developed Least developed

Structural Reduction of HIV stigma, policy reform

Least developed Least developed Least developed Least developed Least developed

Community Mass media, social media, community mobilisation

Moderately developed

Least developed – varies across modes and target groups

Moderately developed Least developed Least developed

Small Group Structured peer based workshops

Most developed

moderate– varies across modes and target groups

Moderately developed Moderately developed

Least developed

Individual Peer and professional counselling

Most developed

Most developed Most developed Most developed Moderately developed

Biomedical Prevention

Increased testing, Treatment as Prevention

Most developed


Least developed Least developed Least developed

Summary of where published evidence about HIV prevention and health promotion is most, moderately or least developed

Recommendations for Improving Evidence BaseTwo key interrelated factors:

• Research: Intervention research trials that use a broad range of rigorous designs applied appropriately to interventions at different levels of health promotion, and investigate what works as well as why it works and in what context.

• Practice: Stronger implementation research within CBOs with a focus on program theory, quality practice indicators, and development of sustainable evaluation and quality improvement approaches that recognise the need to continuously adapt and reorient programs.

Without these reorientations in both research and practice, evidence will =• Limited to the impact of parallel but unconnected strategies • Provide little insight to what are the most effective leverage points, and what to change as the situation evolves.

Where MEL&QI is most, modest and least developed

• Rapid review of current practices used in Australia (building on previous work undertaken by AFAO in 2008)

• Rapid review of evaluation practice in international contexts similar to Australia (primarily Europe and North America).

• Reviewed the published work, abstracts of key conferences attended by HIV educators in Australia and internationally and supplement this with other targeted online searches with organisations. (~reduced to ~100 documents)

• While not a complete audit of all work undertaken - reasonable overview of most key developments in monitoring and evaluation since 2008 with least intrusion on the community sector organisations

Intervention level Strategies (examples) Process and quality practice indicators

Impact Indicators Combination prevention or system wide synergy indicators

Structural Policy and law reform, advisory structures,


Community Community engagement and mobilisation


Online Social Media Least developed Least developed Least developed

Mass media Moderately developed Moderately developed Least developedSmall Group Structured peer based

workshopsModerately developed Moderately developed Least developed

Individual Peer Counselling Models Most developed Moderately developed Moderately developed

Professional Counselling models

Most developed Most developed


Summary of where MEL approaches are most, moderately or least developed

CBO Capacity Building Initiatives

• In broad terms, most programs aimed to move organisations or sectors through stages of evaluation capacity – compliance (fulfilling funding source requirements), – investment (beyond compliance, evaluation is used to improve programs and is supported

by leadership), and – advancement (beyond investment, evaluations are increasingly ambitious and contribute

to prevention theory and practice). (Gilliam et al., 2003)

• Full Report gives examples of Australian and International Initiatives – (incl Acon PEKM)

CBO Capacity Building Initiatives

These and other similar initiatives have generally included among their aims to increase: • capacity to determine why an intervention works, not just if it works,• capacity for continuous quality improvement approaches, and • understanding of, and methods to, identify impact within a combination

prevention or health promotion system,• documenting and sharing of the knowledge and learning.

Monitoring, Evaluation and Learning (MEL) and Quality Improvement (QI) framework

The framework endeavours to acknowledge:• the complexity of the evolving health, social and political systems in which

HIV prevention operates; • the strengths of the partnership response and combination approaches;

and• the rigour of program logic, program theory, quality improvement and

systems thinking.

Priority Community Y

Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral

Individual and clinical focus services

Targeted Community development and social influence

Peer group development and network focused projects

Community Targeted Social marketing

Community venues and settings based engagement

Organisational and systemic change

Structural , policy and social change

Mass Media Social marketing

Population Health Outcome

Reduced transmission

and impact of H

IV

Integrated Combination of Health Promotion Actions and Outputs


Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral











and impact of H

IV

Integrated Combination of Health Promotion Actions and Outputs Across Priority Communities

Integrated Combination of Health Promotion Actions and Outputs









Mass Media Social marketingIn

divi

dual

G

roup

C

omm

unity

S

truc

tura

l



and impact of H

IV










Sector Wide National Strategy outcomes

Longer term / Combined Program Level Impact

Short Term /Individual Project Level Impact


Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral

Improved relevant knowledge, attitudes, skills, and self efficacy

Enhanced quality practice indicators

Indicators of strengthened community capacity and responses

Project level quality and impact indicators

Increased access to health services, testing and treatment

Impact on peer norms and experience

Participation of affected communities

Increased health promoting social norms within priority communities

Increase in levels of protective sexual risk behaviour and testing in at risk groups

Strengthened integration across health promotion strategies

Program level quality and impact indicators

Increased indicators of sustained community responses among priority populations

Increased sustained testing and treatment uptake


and impact of H

IV

Reduced risk behaviours

Decrease undiagnosed HIV

Prevention system level outcomes (linked to National HIV Strategy and Targets)

Improve QoL of PLWHIV

Increase PLWHIV on Treatment with UVL

Strengthened systems in research, evaluation and workforce

Reduced incidence of HIV


Priority Community X Priority Community Y Priority Community Z

Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral




























and impact of H

IV










Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral





..Organisational and systemic change
























and impact of H

IV










Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral





.

.

.Organisational and systemic change
























and impact of H

IV









Inputs/ Resources

Community Organisations and advocacy

Clinical and primary care Services (medical and Counselling)

Advisory structures, Policy, and resource allocation*

External Influences

Social Determinants

Social Drivers

Community capacity , strength and participation

Biomedical testing, treatment and prevention developments

Population impacts of testing and treatments

Social capital

Stigma and discrimination

Partnership*, Governance and Leadership*

Guiding Principles and ethics

Human rights

Research organisations

Inputs/ Resources

Community Organisations and advocacy

Clinical and primary care Services (medical and Counselling)

Advisory structures, Policy, and resource allocation*

External Influences

Social Determinants

Social Drivers

Community capacity , strength and participation

Biomedical testing, treatment and prevention developments

Population impacts of testing and treatments

Social capital

Stigma and discrimination

Individual and interpersonal theories

Structural and System theory

Social / Behavioural theories

Social and Epidemiological Research

Project, Program and system level evidence and evaluation

Continuous Quality Improvement, refinement of practice guidelines and standards, and development of workforce*

Partnership*, Governance and Leadership*

Information Systems* (Monitoring, Evaluation and Learning)


Indi

vidu

al

Gro

up

Com

mun

ity

Str

uctu

ral

*Prevention System Strengthening building blocks identified by WHO





.

.

.Organisational and systemic change
























and impact of H

IV









International practice and developments

Guiding Principles and ethics

Human rights

Research organisations

Example inputs and resources

Example Project

Example indicators for Project level MEL&QI (preferably drawn from project’s own project logic)

Example Program Level MEL&QI (such as range of peer based projects)

Example quality practice indicators

Example Indicators of Inter-project quality links

Example Project Output Indicators

Example Project level impact indicators. (immediate to 3 month)

Example Program level quality indicators

Example Inter- program quality links

Program level impact indicators(3 to 12 month)

Community organisation resources Principles of peer based programs Peer based staff and volunteers Evaluation from previous programs

Small Group Level Project: eg- Peer Group workshop for gay men

Quality practice involvement of target group in development and improvement. Satisfaction measures Group interaction and dynamics indicators Evidence of reciprocal learning between participants Proportion of participants who complete workshops Peer referrals / recommendations

Referrals from outreach, online initiatives, counselling Discussion or use of social marketing campaign within workshop Community volunteer engagement indicators

Number of workshops conducted Average number of participants completing workshops Alignment of intended target group and activity participants

A workshop would be focused on only three or four of a set of project level impact indicators – depending on the focus of the workshop. The following is an example of a set of indicators from which a workshop may draw: 1. Increase in sexuality related health literacy and support seeking knowledge.2. Increase in knowledge and confidence to interact in diverse and sexualised environments (eg online, SOPV, etc).3. Increase in skills and confidence to negotiate sexual interactions including safe sex practices 4. Increase in confidence to manage HIV disclosure in sexual and social settings5. Increase in knowledge and confidence regarding sexual technique and repertoire6. Increase in confidence to develop relationships (intimate and friendship).7. Increase in indicators of participants influencing their peers regarding peer program messages

Indicators of participants influencing their peers in relation to program aims Increased indicators of sustained community responses among priority populations Indicators of community level engagement with strategies Volunteer recruitment from peer programs

Strategic links between peer group project and community development projects Strengthened integration and strategic links across peer based programs and other promotion strategies Referrals to and from venue outreach, online initiatives, or counselling

Increased health promoting social norms within priority communities Indicators of testing and treatment uptake Increase in levels of protective sexual risk behaviour and testing among program participants

Application of MEL&QI framework to a hypothetical peer group workshop for gay men

Project level quality, monitoring and evaluation

Program Level quality, monitoring and evaluation

Prevention system level quality, monitoring, surveillance and evaluation

Project /Service staff Yes Possibly NoAgency/Program Yes Yes PossiblyExternal evaluators Possibly Yes PossiblyHealth Services Data Possibly Yes YesEpidemiology and Social Research Centres / Department

No Possibly Yes

Guidelines for responsibility for collecting and summarising MEL data

Final Comments

This is a draft and at a conceptual level– Draft summary and full report available for comment – Presented as a discussion monograph in July

Possibly more than ever our community sector needs to • look at frameworks and approaches to building and expanding the evidence base,

particularly where it is less developed • Recognise that the projects and programs will continuously evolve and change• Understanding the what, why, and in what combination or system of approaches• Turning this into a useable shared evidence base

Health & Medicine

A draft report of the Evidence Synthesis and Application for Policy and Practice project