• Evaluation is a professional and ethical responsibility and is a core part of PHN professional practice

• Commitment to evaluation helps build the PHN intelligence about what interventions work in what context why and how

• Evaluation findings helps improve practice to be more effective and efficient

EvaluationProcess Evaluation

Process Evaluation

Process Evaluation

Process Evaluation

Evaluation• Evaluation determines the extent to which an intervention has achieved

the desired health outcomes and assesses the contribution of strategies used

• Key reasons for evaluation PHN interventions is to assess and improve intervention: Effectiveness – has the intervention worked Efficiency – relative effectiveness compared to other interventions Efficacy – effectiveness under ideal circumstances Economic Impact – cost-effectiveness and use of resources Intelligence – inform future planning and theory building Accountability – justify resource allocation and use

Process Evaluation

Relationship of goals and objectives to evaluation

Problem reflected in Goal measured by Outcome evaluation

Determinants

reflected in Objective measured by Impact evaluation

Strategies reflected in Strategy Activities

measured by Process evaluation

Action Statements and Evaluation Levels

Process Evaluation

Levels of Evaluation• There are several different levels of evaluation in PHN practice:

1. Formative Evaluation - data collected prior to intervention implementation which is used to inform intervention design and assess capacity

2. Process Evaluation – assessed the intervention strategies and capacity building strategies

3. Impact Evaluation – measures whether the intervention objectives have been met

4. Outcome Evaluation – measures whether the intervention goal has been met

5. Economic Evaluation – measures cost-effectiveness of the intervention or intervention strategies

Process Evaluation

Qualitative V’s Quantitative methods• The two main forms of data gathering used in evaluation include:

qualitative and quantitative

• Quantitative methods focus on numeric data that can be statistically analysed and can test the extent to which an intervention causes change in health status, health behaviour, knowledge, attitude etc

• Qualitative methods attempt to determine the meaning and experience of the intervention for the target group and other participants

• Good quality evaluation usually has components of both qualitative and quantitative methods

Process Evaluation

Process Evaluation• Process evaluation assesses intervention implementation and is

concerned with • Intervention exposure – extent target group are engaged or aware of PHN

problem• Reach – proportion of target group who participate• Participant satisfaction – whether participants are happy and like the

intervention activities• Delivery – whether activities are implemented as intended• Fidelity – assessing performance of intervention materials and components• Contextual aspects – aspects of the environments that influence the

intervention implementation

Process Evaluation

Process Evaluation

• Both quantitative and qualitative methods are used in process evaluation: Quantitative methods measure reach, delivery and exposure

aspects of the intervention Qualitative methods assess participant satisfaction, fidelity and

context elements of intervention delivery.

• Process evaluation provides rapid feedback on the quality and integrity of the intervention – useful management tool

• Process evaluation is relatively low cost and is a useful quality assurance tool

Process Evaluation

Evaluating education materials

• Several tools exist for evaluating education materials:

Standard protocol for leaflets and audiovisual materials- considers: attraction, comprehension, acceptability, personal involvement,

persuasion

SMOG test- Formula for readability by calculating the number of polysyllabic words

Group leader performance- true/false questionnaire completed by participants

Process Evaluation

Methodological component

General definition Example – qualitative and quantitative methods

Design Timing of data collection: when and how often data will be collected

Observe classroom activities at least twice per semester with at least 2 weeks of observation Conduct focus groups with participants in the last month of the intervention

Data sources Source of information (for example, who will be surveyed, observed, interviewed)

Both qualitative and quantitative – data sources include participants, teachers/staff delivering sessions records, the environment etc

Data collection tools/ measures

Instruments, tools and guides used for gathering process-evaluation data

Both qualitative and quantitative – tools include surveys, checklists, observations forms, interview guides etc

Data collection procedures

Protocols for how the data collection tool will be administered

Detailed description of how to do quantitative/ qualitative classroom observation, face-to-face or phone interview, mailed survey, focus group etc

Data management

Procedures for getting data from field and entered – plus quality checks

Staff turn in participant sheets weekly, evaluation coordinator collects and checks surveys and gives them to data entry staffInterviews transcribed and tapes submitted at the end of the month

Data analysis Statistical and/or qualitative methods used to analyse or summarise data

Statistical analysis and software that will be used to analyse the quantitative dataTypes of qualitative analysis used

Key methodological components to consider in process evaluation

Process Evaluation

Process indicators

• Evaluation indicators are the criteria against with data or observations are assessed for judgement of intervention success or failure

• Evaluation indicators may come from : Historical comparisons with similar efforts in the past Comparisons with contemporary activities Professional consensus – using the above and professional

judgement

• Finding comparison data may be difficult due to a lack of published results hence collective professional judgement should be applied