How to Mon tor for Populat on Health Outcomes: Gu del nes ...€¦ · by analysing population health outcomes and risks and determinants, and examining inequalities across regional

� How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

How to Mon�tor for Populat�on Health Outcomes: Gu�del�nes for develop�ng a mon�tor�ng framework

Public Health Intelligence Occasional Bulletin No. 44

��

–

How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

AuthorDr John Wren from the Public Health Intelligence (PHI) Unit

in the Ministry of Health wrote this guide.

Citation: Ministry of Health. 2007. How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework. Occasional Bulletin No. 44.

Wellington: Ministry of Health.

Published in July 2007 by the Ministry of Health

PO Box 5013, Wellington, New Zealand

ISBN 978-0-478-19150-9 (print) ISBN 978-0-478-19153-0 (Internet)

HP 4414

This document is available on the Ministry of Health website: http://www.moh.govt.nz

�� How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

ForewordPublic Health Intelligence (PHI) is the epidemiology group of the Ministry of Health. PHI carries out the Ministry’s statutory responsibility to monitor the health of the New Zealand population by analysing population health outcomes and risks and determinants, and examining inequalities across regional boundaries and between population groups. An important role for PHI is the delivery and dissemination of epidemiology evidence for the development of policy and decision-making in the health sector.

How to Monitor for Population Health Outcomes presents guidance to public and population health programme managers, and interested others, on how to develop indicators to monitor progress on achieving population health outcomes sought from their programmes. This guideline has been designed to complement the information and guidance presented in the Ministry’s Guide to Developing Public Health Programmes: A generic programme logic model (Ministry of Health 2006). The need to develop a robust outcomes monitoring framework is driven by an increased expectation from central government that there will be a focus on results in the design and delivery of publicly funded services. This expectation is highlighted by the requirements of the Public Finance Act 2004 and the Crown Entities Act 20041 (The Treasury and State Services Commission 2007a; b).

We welcome your comments and suggestions about the contents, and any additions or clarifications you might have.

Barry BormanManager (Epidemiologist)Public Health Intelligence

1 Crown entities includes District Health Boards.

�v How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

AcknowledgementsThe author wishes to particularly thank:

• Geoff Stone, Senior Advisor, Centre for Social Research and Evaluation, Ministry of Social Development, for his contribution to the development of definitions in the Glossary and insights into outcomes monitoring in social settings other than public health

• Craig Wright, Senior Advisor (Statistics), Public Health Intelligence (PHI), Ministry of Health, for his significant preparation in Part C of the discussion on the small numbers problem and the statistical ability to detect change in health outcomes of interest; and

• Dyfed Thomas and Sarah Gerritsen, (PHI), for their preparation of Part D.

The author also wishes to thank the following external peer reviewers for their insightful comments:

• Anne Dowden, Director – Evaluation, Research New Zealand

• Rob Smith and Tony Walzl, Allen and Clarke Policy and Regulatory Specialists

• Ray Prebble, Editor, Macmillan and Prebble.

The following people provided valuable internal peer review:

• Dr Chris Wong, Public Health Physician, Ministry of Health

• Sarah Gerritsen, Senior Advisor (Population Health Research), PHI

• Dr Kirstin Lindberg, Senior Advisor (Public Health Medicine), PHI.

v How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

ContentsForeword ................................................................................................................................��

Acknowledgements.................................................................................................................�v

Part A: Introduct�on ................................................................................................................ 1

How to use this guide ........................................................................................................... 1

Why measure performance? ................................................................................................. 2

Part B: What Object�ves and Assoc�ated Outcomes Need to be Mon�tored? .............................. 3

Introduction: outcomes monitoring programmes .................................................................. 4

What should you measure to demonstrate the success of the programme? ........................... 4

What makes good outcome statements that are measurable? ............................................... 6

Have you used action words in active statements? ........................................................... 6

Are they SMART objectives and/or outcomes? .................................................................. 7

Monitoring double-barrelled objectives ............................................................................ 8

Prioritising objectives/outcomes for measurement .............................................................. 8

Management selection criteria for prioritising what is essential to monitor in public health programmes ............................................................................................. 10

Instrument selection criteria for prioritising what is possible to monitor appropriately in public health programmes ......................................................................................... 12

Additional questions and guidance for selecting outcome measures and indicators ........... 13

Do you need a new outcome indicator or instrument? .................................................... 13

Match the measurement instrument to the objective ...................................................... 14

Cost-effectiveness measures .......................................................................................... 15

Decide on the type of instrument ................................................................................... 15

Collecting and analysing information .................................................................................. 16

Part C: Issues to Cons�der �n Outcomes Mon�tor�ng ............................................................... 17

Outcomes monitoring as a tool for ‘continuous programme improvement’: the problem of accountability, attribution and performance management ................................................. 17

Outcomes monitoring versus evaluation ............................................................................ 19

The problem of small numbers ........................................................................................... 19

Detecting change in small numbers ............................................................................... 20

v� How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

Part D: Ava�lable Data and Informat�on from Publ�c Health Intell�gence .................................. 24

Survey data ........................................................................................................................ 24

Administrative data ............................................................................................................ 26

PHIOnline – www.phionline.moh.govt.nz ............................................................................ 27

Glossary ............................................................................................................................... 30

References ........................................................................................................................... 44

Further reading ................................................................................................................... 45

F�gures:Figure 1: A framework for prioritising what outcomes to monitor ................................................ 9

Figure 2: Simulated effect of 15% reduction in violent offences ............................................... 22

Figure 3: Power to detect change due to an intervention, for specified incidence, population size and 10% incidence reduction ......................................................................... 23

Figure 4: The PHIOnline home page ........................................................................................ 27

Figure 5: PHIOnline single map interface ................................................................................. 28

Figure 6: PHIOnline double map interface ............................................................................... 29

Tables:Table 1: Distinguishing objectives, outcomes and outputs ....................................................... 5

Table 2: List of action words for writing performance objectives................................................. 6

Table 3: Defining SMART objectives ........................................................................................... 7

Table 4: Checklist and score card for prioritising outcomes and indicators for monitoring ........ 13

Table 5: Summary of New Zealand Health Monitor surveys, 2002–2012 .................................. 25

Table 6: Summary of administrative data sources .................................................................... 26

1 How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

Part A: Introduct�on

Key Po�nts• This How to aims to help public and population health programme managers to develop

timely, appropriate and cost-effective outcome measures for programmes funded by government.

• Measuring programme performance is essential for maintaining and enhancing support for public health programmes.

• The process of developing a logic model and an outcomes monitoring framework is intended to facilitate a process of ‘continuous programme improvement’.

• This How to should be read in conjunction with A Guide to Developing Public Health Programmes: A generic programme logic model (Ministry of Health 2006).

How to use th�s gu�deThis How to aims to provide guidance to public and population health programme managers and planners – whether they be in District Health Boards, non-governmental organisations or government departments – on how to select timely, appropriate and cost-effective outcome measures for public health programmes.

Throughout this How to we aim to provide plain English advice, checklists and examples to guide you through the process of selecting which outcomes to measure and how they should be measured. We provide information about some issues that are likely to arise from adopting a more rigorous outcomes-focused framework to monitoring public health programmes. We also outline the types of data held by the Ministry of Health that may be freely used to help monitor the performance of your public health programmes.

The guide is structured into five parts.

• Part A: Introduct�on outlines the purpose of this document and where it fits in relation to other Public Health Intelligence (PHI) publications.

• Part B: What Object�ves and Assoc�ated Outcomes Need to be Mon�tored? describes a process for translating policy goals and objectives into measurable outcome statements, and a process for selecting and prioritising the outcomes to be monitored.

• Part C: Issues to Cons�der �n Outcomes Mon�tor�ng discusses a number of key issues arising from the application of an outcomes monitoring approach to publicly funded programmes and small populations.

• Part D: Ava�lable Data and Informat�on from Publ�c Health Intell�gence introduces PHIOnline, a free public access internet-based service to a wide range of government statistics that may be useful in monitoring a population health programme.

• A Glossary provides an alphabetical list of terms, definitions and explanatory text relating to outcomes monitoring language.


This How to builds on the material presented in A Guide to Developing Public Health Programmes: A generic programme logic model (Ministry of Health 2006). The Guide provides a general introduction on how to design and implement comprehensive and measurable public health programmes in New Zealand, and includes some material on designing outcome measures (Ministry of Health 2006). This How to significantly expands on that material, and material presented at a PHI analytical workshop on Monitoring Public Health Outcomes held in October 2006. Ideally the Guide should be read before this How to.

The Ministry of Health’s Leading for Outcomes website http://www.leadingforoutcomes.org.nz/ is another Ministry-led initiative that promotes and uses an outcomes framework to improve how we think about and improve health in New Zealand, starting with cardiovascular disease (CVD) and diabetes. The website sets out a model of risk factor and disease progression, an outcomes hierarchy and an indicators framework for measuring progress towards achieving better CVD and diabetes health outcomes in New Zealand, with a view to extension of the approach to other health and disability areas.

Why measure performance?The main motivation for introducing management tools such as programme logic models, outcomes monitoring and managing for results is an increased expectation from central government that the design and delivery of publicly funded services will have a strong results focus. This expectation is highlighted by the requirements of the Public Finance Act 2004 and the Crown Entities Act 20042 and associated guidance documents (The Treasury and State Services Commission 2007a; b) for the development of statements of intent by departments and Crown agencies.

To sum up, measuring performance:

• facilitates change and improvement

• is a mechanism for accountability

• supports planning and decision-making relating to resources

• can highlight areas requiring further work.

Ideally, performance measures for public health programmes should provide information about:

• a change in health status and health determinants achieved in priority population groups, including changes in inequalities

• resource and service utilisation

• the programme’s responsiveness to the target population.



Part B: What Object�ves and Assoc�ated Outcomes Need to be Mon�tored?

Key Po�nts1. It is not always possible to monitor every objective and outcome, so you will need to

establish a list of prioritised objectives/outcomes for monitoring.

2. The process of planning an outcomes monitoring framework begins with translating policy goals and objectives into SMART objectives:

• Specific

• Measurable

• Achievable (sometimes ‘Accurate’ or ‘Action-oriented’ are used)

• Relevant (sometimes ‘Realistic’ is used)

• Time-based.

3. Management and instrument criteria can be used to assess whether it is possible and essential to monitor the objective or outcome of interest. Management selection criteria are:

• attribution (accountability)

• centrality

• cost-benefit

• robustness to withstand public scrutiny

• timing.

Instrument selection criteria are:

• availability

• reliability

• sensitivity

• validity.

4. Planning for outcomes monitoring requires decisions about:

• What information is required? What is the data going to be used for?

• What data is already available? Is new data really needed?

• Who is going to collect the data?

• What type of measurement instrument could or should be used?

• How many measurement instruments are to be used?

• Who is going to do the analysis?

• When is the data required by to inform timely decision-making?

5. Table 4 provides a checklist and scorecard to help prioritise which objectives and outcomes are essential to monitor.

6. See the Glossary for a full list of terms and associated definitions used throughout this How to, and extra explanatory comment and examples illustrating the use of the terms.


Introduct�on: outcomes mon�tor�ng programmesEstablishing which outcomes need to be – and can be – successfully monitored requires careful thought and planning. Preparing an outcomes monitoring plan will help you to establish a successful monitoring programme.

An outcomes monitoring plan is a process for the routine systematic collection and recording of timely information about aspects of a programme to assess whether progress is being made on achieving the programme’s objectives, and how the programme could be improved. Information is timely when it contains data directly relevant to the programme and – importantly – is available in a timeframe where it can usefully inform decisions about whether the programme is performing as planned, and what things need to be changed to improve the performance of the programme if required.

Collecting baseline data at the start of a programme is vital to be able to assess what change has occurred over time, particularly when you are trying to attribute a change to an effect of the intervention.

In preparing the outcomes monitoring plan you will need to assess:

• what needs to be measured to demonstrate success

• how things should be measured and by whom (deciding what indicators are going to be used, after considering issues such as validity, reliability, sensitivity, attribution, availability of suitable instruments, cost-benefit)

• the timeframes for delivering information about the performance of the programme against the stated objectives to decision-makers and key stakeholders

• whether a formal evaluation process is required.

A good outcomes monitoring process:

• identifies and prioritises the outcomes that are essential to monitor and that can be monitored appropriately

• provides timely information to key decision-makers and stakeholders about the progress made on achieving the desired outcomes

• uses outcome measures robust enough to withstand public scrutiny

• is cost effective

• includes baseline data relevant to the intervention.

What should you measure to demonstrate the success of the programme?Critical to any successful outcomes monitoring plan is to identify what could and should be measured in order to show that the programme is being implemented as planned, and that progress is being made to achieve the desired health outcome(s). The steps involved are:

1. identify what could be measured

2. prioritise these into those that should be measured if resources allow

3. identify the essential few that must be monitored.


Underpinning this approach is the assumption that it is not always feasible – or necessary – to measure everything.

The first place to look to determine what could and should be monitored is to identify what the key programme objectives, outcomes and outputs are. You should be able to identify these from the ‘logic model’ developed for the programme. A logic model is a planning tool that describes the purpose, what, where, when, and how of the programme being implemented.

Ideally, programme objectives, outcomes and outputs should be clearly distinguished from each other (see Table 1 for definitions of these terms) and expressed in action words. Expressing outcomes statements clearly makes it easier to identify what actions need to be undertaken, what changes need to take place, what the desired end result is, and consequently what needs to be monitored.

Table 1: D�st�ngu�sh�ng object�ves, outcomes and outputs3

Object�ves are statements about the results a programme seeks to achieve. Any programme must have at least one objective.

Objectives may form a hierarchy that moves from a limited set of high-order objectives that are synonymous with aims or goals to be achieved in the long term (five to seven years). Underneath high-level objectives are more intermediate-level objectives that are to be achieved in a three- to five-year time frame, and which must be achieved in order to attain the high-level objective. The lowest levels of objectives are immediate or operational objectives that must be achieved first – typically in one or two years.

Objectives may be translated directly into ‘outcomes’ if they deal with only one issue. However, double-barrelled objectives will require multiple outcome measures to be developed.

Outcomes are specific statements about the intended change in public health-related attitudes, knowledge, behaviours, or physical health status in the target population(s) sought by undertaking the planned public health activity. In some situations ‘process’ outcomes may be desirable.

Process outcomes typically measure the amount of effort put into a programme and the quality of the service provided. They can also be appropriate where it is important to monitor community support for a programme. Process outcomes that measure effort can be expressed as ‘outputs’.

Outputs are things (such as goods) produced, services delivered, events held, or participation generated resulting from the activities undertaken.

3 See the Glossary for a full list of other terms and associated definitions used throughout this How to, and extra explanatory comment and examples illustrating the use of the terms.


What makes good outcome statements that are measurable?Have you used act�on words �n act�ve statements?In some cases, even though a programme’s objectives and outcomes can be identified, they may be stated in a way that makes them difficult to measure or unsuitable for programme planning or contracting purposes. This can happen because the language used is too passive in tone, or is too complex or abstract. Consequently, in some situations it may be necessary to translate the identified objectives into language that is more suitable for monitoring. Ensuring the language used is active rather than passive will help to do this.

Active statements involve ‘action’ words, and a list of action words identified as suitable for public health is provided in Table 2.

Table 2: L�st of act�on words for wr�t�ng performance object�ves

Accept Adopt Advocate Analyse Arrange

Approve Appraise Bargain Calculate Care

Change Choose Classify Categorise Challenge

Chart Compare Conduct Construct Contrast

Co-operate Check Defend Define Demonstrate

Describe Develop Differentiate Discriminate Draw

Evaluate Execute Explain Express Fill out

Forecast Formulate Generate Identify Inform

Instal Interview Judge Justify Label

List Locate Manipulate Modify Name

Operate Organise Outline Persuade Plan

Prepare Prescribe Produce Purchase Question

Rank Recall Recognise Reflect Remove

Research Resolve Review Select Sort

Specify State Study Take Tell

Translate Use Write

Source: Bartholomew et al 2006

Take, for example, the following objective:

Communication plans will be developed to ensure stakeholders are kept informed.

This sentence contains the idea of what needs to be done, but responsibility for doing it and therefore an emphasis on the action needed is lost because of the passive form. Compare this with the alternative wording, using an active form and action words:

The committee will develop communication plans to ensure stakeholders are kept informed.


Using a bullet list for objectives lets you start each objective with an action word, which can be a forceful way of conveying the objectives; for example:

The committee will:

• develop communication plans to ensure stakeholders are kept informed

• prepare a report on the nutrition in schools open day

• inform parents about the progress their children have made.

Are they SMART object�ves and/or outcomes?Another good approach to identifying which objectives and/or outcomes are able to be measured, or to reconfigure them so that they are measurable, is to ask yourself, are they SMART objectives (Iverson 2003)? SMART objectives/outcomes are defined in Table 3.

Table 3: Defin�ng SMART object�ves

Item Definition

Specific An objective should address a specific target or accomplishment. Specific implies that

an observable action, behaviour or achievement is described, which is also typically

linked to an identifiable change in rate, number, percentage or frequency.

Measurable A method should be established to indicate that an objective has been met. That is,

there should be a system, method or procedure for tracking and recording the change in

behaviour or action towards which the objective is directed.

Achievable* Though not necessarily easy or simple, the objective should be feasible – that is,

capable of being achieved. Objectives should be limited to what can realistically be

done with available resources, and ideally the resulting change should be ‘attributable’

to the action undertaken.

Relevant* An objective should be significant to the people involved in the programme (from

beneficiaries to the programme’s sponsoring organisation), and the objectives should

be capable of having an impact or making a change.

Time-based An objective should be achievable within a specific timeframe. Generally this takes the

form of a start and end date. The time may be short (two or three months, up to two

years), medium (three to five years), or long term (five years to seven years).

Source: Iverson A. 2003. Preparing Program Objectives: Theory and Practice. The International Development Research Centre: Evaluation Unit, Toronto

* What A and R stand for is inconsistent in the literature. A is sometimes given as ‘accurate’, ‘action-oriented’, ‘accountable’ or ‘attributable’. R is sometimes given as ‘relevant’ or ‘realistic’. In some situations, these alternative phrases may be more useful, in which case use the term that is most appropriate or that most clearly helps to describe the intent of the objective.


Mon�tor�ng double-barrelled object�vesDouble-barrelled objectives contain multiple components, and if well written have the advantage of being able to convey multiple objectives in one single sentence rather than using several sentences. However, they are also more complex to monitor. Using the above criteria, we need to ask the question: Are double-barrelled objectives SMART? Here are two examples of double-barrelled objectives.

• Objective 1: Enable people with chronic conditions to improve their health, slow progress of their condition(s), and maintain independence wherever possible by aligning community and hospital services across [ ] DHB.

• Objective 2: Reduce the incidence of cancer, diabetes, and cardiovascular disease by 20% over the next five years.

The first objective has multiple components in that it refers to chronic conditions, and talks about slowing the progress of the conditions, maintaining independence and aligning services. It could be argued that this objective, as written, is a good high-level objective in that it uses some action words that provide an overall general direction for the DHB, and it complies adequately with the SMART criteria in Table 3 in terms of a high level objective statement.

However, for operational and monitoring purposes it is does not adequately comply because it does not specify the chronic conditions, time frame and services. All of these issues would have to be addressed when developing the logic model and outcomes monitoring plan to achieve this high-order objective. Multiple outcome measures would have to be used to assess whether this objective was achieved, because different measures would need to be used for each component.

The second objective also has multiple components, but it clearly specifies the chronic conditions of interest, the size of change sought, and the timeframe for achieving the changes. The objective complies with the criteria in Table 3, although the objective would also require the use of multiple outcome measures – a different measure for each chronic condition – for progress on its achievement to be monitored appropriately.

In answer to the question, are double-barrelled objectives SMART? the best advice is generally to avoid double-barrelled objectives when writing low-order (ie, operational) objectives. This will make planning clearer and the objectives easier to monitor. Where double-barrelled objectives may be desirable (eg, when writing high-level objectives), make sure they comply as closely as possible with the criteria in Table 3. Also, remember that where double-barrelled objectives are used, multiple outcome measures will generally have to be developed to monitor each component of the objective statement.

Pr�or�t�s�ng object�ves/outcomes for measurement After identifying and clarifying the range of objectives and/or outcomes that could be monitored in an ideal world, it is highly likely that it will not be feasible – or necessary – to monitor all the objectives and outcomes planned for. Consequently, a process of prioritisation will have to take place to select the objectives and outcomes that are both essential and possible to monitor.


We suggest two types of criterion to use as aids in making decisions about which outcomes should be and can be monitored:

• management considerations

• instrument selection considerations.

Management considerations are concerned with promoting the development of an appropriate and robust monitoring system. Instrument selection considerations are concerned with establishing scientific credibility for the monitoring system. Figure 1 illustrates how these two types of criterion relate to each other. They are also further discussed in the following pages. Although the criteria can be defined separately, in practice they are interrelated and a judgement will have to be made as to where the balance lies between the merits of each.

F�gure 1: A framework for pr�or�t�s�ng what outcomes to mon�tor

By examining each objective and indicator using the criteria set out below, you will be able to develop a prioritised selection of the outcomes and associated indicators that can and should be measured appropriately. To help with this, it may be useful to use a scorecard such as that provided in Table 4. The scorecard provides a method for rating each outcome and indicator against the criteria. Note that more than one indicator can be used to monitor an outcome, but it may not be necessary or cost effective to do so.

The scorecard is simple to use: just assign a ‘1’ score to each criterion that has been successfully met. All the scores are added up, and the outcomes and associated indicators with the highest score should receive the highest priority for inclusion in the outcomes monitoring plan. A slightly more complex approach, which would provide better differentiation between possible indicators, would be to use a scoring approach that allows for decimal points; for example, instead of scores of 1 or 2, a score of 1.5 could be allocated. In some situations a more sophisticated approach may be desirable, in which case a ‘weighted’ system could be applied to the scorecard. In a ‘weighted’ approach, some criteria would be judged as more or less important than others, and consequently a higher or lower range of possible scores could be allocated to the selected criteria. For example, because of the size and nature of the programme it may be decided that the instrument selection criteria of validity and reliability are so important that they are worth double points compared to the others (ie, the results are going to be weighted (biased) towards indicators that score well on those factors).

Management criteriafor selecting outcomesthat should bemonitored:• attribution (accountability)• centrality• cost–benefit• robustness to withstand scrutiny• timing.

Instrument selectioncriteria for selectingoutcomes that can bemonitored robustly:• availability• reliability• sensitivity• validity.

Outcomesprioritised

formonitoring


Management select�on cr�ter�a for pr�or�t�s�ng what �s essent�al to mon�tor �n publ�c health programmesAttr�but�on (accountab�l�ty)Attribution is the extent to which change in the outcome of interest is associated with the type of activity undertaken. Attribution is an important issue to consider because it has implications for how well a programme’s activities can be said to have resulted in any changes observed, and for assigning the level of accountability for the success or failure of a programme or its components.

Strong attribution requires being able to establish a clear and unambiguous causal link between what you do and what happens – something that is often problematic for many public health programmes. When assigning attribution, you need to beware of any ‘attributional bias’ that results from over-attributing a change to any particular activity.

Central�tyHow important is the outcome of interest to establishing the success of the programme? The more central the outcome, the more important it is to measure it appropriately. Where data is not available, think carefully about developing a new measure, or using an appropriate ‘proxy’ measurement (see ‘Availability’ below).

Whether a programme is deemed successful or not is not always related to whether the central health outcome of interest has been achieved. A programme may fail in one aspect but still be successful if other outcomes of importance have been achieved.

Cost–benefit‘Cost–benefit’ refers to the balance between the cost of using and/or developing a measure and the benefit that will be gained from implementing it. For many programmes, you will need to make a judgement about whether the cost of developing a monitoring regime or measure is worth the benefit to be gained from the information provided. Where the cost outweighs the benefit, then consider using an appropriate proxy indicator, or information from a pre-existing monitoring regime.

In general, the proportion of the total budget spent on monitoring should be in the region of 5% to 20%, depending on the type and size of the programme. If the programme is a pilot initiative, the results of which could be used to significantly influence whether the programme is expanded into a major effort, then 20% of the total budget could be appropriate. If the programme has a large budget and is well established, applied in standard ways and supported by evidence for its effectiveness, then a budget allocation of 5% may be more appropriate for monitoring purposes.

Cost per output is probably the best method to use when calculating the cost of a monitoring programme or a component of it. For example, the cost of purchasing information from an existing data set may be cheaper than doing it yourself. Or, it may be cheaper to use a proxy measure than to use a direct measure (assuming the proxy measure is robust enough to withstand public scrutiny and the information trade-off is acceptable). Note that cost effectiveness is just one of a number of criteria that should be used in planning an outcomes monitoring regime.


Robustness to w�thstand scrut�nyRobustness will be provided by ensuring that the instrument selection criteria (presented below) are observed. Be very clear about why a particular set of outcomes has been selected for monitoring and not others, and why particular measurement instruments have been selected over others. Ask yourself the following.

• What is being monitored?

• Why and how is it going to help to assess the effectiveness or efficiency of the programme as a whole?

• Why is it essential?

• When must the information be available?

• How is the data to be collected?

• Who is going to do the collection and analysis?

• Is the proposed approach cost effective?

• Will the information be provided in a timely manner?

• Will the monitoring system withstand scrutiny by stakeholders?

For public health programmes funded by government money, it is important that assessments of the effectiveness of the programme be able to withstand public scrutiny. In this context, we suggest it is usually advisable not to attribute a programme’s success to a single criterion: robustness is provided by the strength of logic of the total outcomes monitoring regime.

T�m�ngTiming has two aspects to it. The first concerns establishing ‘attribution’ (see above). In this case, timing refers to establishing the time sequence between when an activity took place and when a change in the desired outcome was observed. The second aspect relates to being able to report to stakeholders on the performance of a programme in a timely manner, which also means the proposed indicator must be able to be produced within an appropriate timeframe.

As a general rule, the timeframe for short-term objectives can be as short as two to three months or up to two years. The medium term is defined as three to five years, and a timeframe of five to seven years is usual for achieving long-term objectives (The Treasury and State Services Commission 2007a; b).

In practical terms, it may not be feasible to monitor/measure progress on attaining short-term, or even medium-term, objectives using national data. For example, information from national survey and administrative data sets is typically not available anywhere from 6 to 36 months after the data was originally collected due to data checking and quality control processes. This may mean that it is not feasible to use information from these data sources as indicators because the data is not available soon enough.


Instrument select�on cr�ter�a for pr�or�t�s�ng what �s poss�ble to mon�tor appropr�ately �n publ�c health programmesAva�lab�l�tyThis refers to whether data, or a measuring instrument, already exists at the local or national level that could be used to help monitor the outcome of interest. Where possible, use existing data that may be sourced from within your organisation or a range of government agencies or organisations such as public health organisations in your area. A good place to start seeing what data is already available to help monitor your programme is at PHIOnline, www.phionline.moh.govt.nz, which includes data at national, District Health Board and Territorial Local Authority level (see Part D for an introduction to the range of information available to you on PHIOnline). Using existing information sources can save a considerable amount of time and effort.

Where such data exists, efforts should be made to use that data or instrument rather than invest in developing new data sources or instruments. If issues such as cost, complexity and/or timeliness prevent direct measurement of the outcome of interest, consider using a proxy indicator. If achieving the central outcome is vital, and data is not readily available, careful thought needs to be given to spending extra effort on developing a new measure, or on identifying a suitable proxy measure.

Rel�ab�l�tyReliability is the extent to which a measure, when used repeatedly in the same way, will produce the same or a similar result. For example, if you were interested in monitoring a person’s weight and used a weighing machine that produced a different reading each time it was used (when all other factors were the same), then the machine would not be a reliable instrument to use.

Sens�t�v�tySensitivity refers to how well a measure is able to detect when a change has occurred in the outcome being monitored. For example, a weighing machine that was able to differentiate a change in weight by 100 gram graduations is more sensitive than a machine that can only detect changes in 500 gram amounts.

Specificity is another criterion that is particularly appropriate in clinical settings, where it is used to assess how accurate a clinical test is in identifying people at risk.

Val�d�ty Validity refers to how well a measurement, index or indicator reflects the outcome it is intended to measure. For an outcome measure to be valid, it should be both sensitive and specific.

Often there is more than one way of measuring an outcome, and the most valid measure may not be practical in all situations. For example, the best way to measure body composition (particularly lean and fat mass) is under-water densitometry (weighing), but this is not practical outside small clinical studies. Therefore, measures of weight adjusted for height (eg, body mass index, BMI) are often used to reflect body composition, particularly body fat mass. Although BMI is correlated with body fat mass, this relationship varies according to body build, age and ethnicity. Therefore, BMI is not recommended for assessing excess body fat mass (obesity) at an individual level.


However, BMI can be useful for assessing obesity at a population level if used and interpreted appropriately. BMI based on direct measurement of height and weight is more valid than BMI based on self-reported height and weight, particularly if measurements are taken by trained observers using appropriate equipment and standardised procedures.

Ultimately, the choice of measure for a particular programme will depend on a number of factors, including validity and practicality.

Table 4: Checkl�st and score card for pr�or�t�s�ng outcomes and �nd�cators for mon�tor�ng

Outcome of �nterest

Ind�cator name

Cr�ter�a Total score

Pr�or�ty

At Av Ce Cb Re Rb S T V

Outcome 1

Output 1

Objective 1

Cr�ter�a abbrev�at�ons

At Attribution (Accountability) Av Availability Ce Centrality

Cb Cost–benefit Re Reliability Rb Robustness to withstand public scrutiny

S Sensitivity T Timing V Validity

Source: Adapted from Table 1 ‘How to select indicators’ (page 68) in United Nations Development Programme Evaluation Office 2002

Add�t�onal quest�ons and gu�dance for select�ng outcome measures and �nd�cators Do you need a new outcome �nd�cator or �nstrument? Think carefully before developing a new outcome indicator or instrument, ask yourself the following.

• Why is a new indicator/instrument needed?

• Is it because there is a problem with an existing indicator/instrument, or is it because it really is a new outcome?

• Can we use existing data creatively to answer the question?

Every instrument has its own strengths, weaknesses and peculiarities – the perfect instrument does not exist. If you are unfamiliar with the instrument that best suits your plans, seek help from someone who knows about it already. Public Health Intelligence at the Ministry of Health have extensive expertise in measuring public health outcomes. If in doubt, seek their advice.


Match the measurement �nstrument to the object�veChoose the level of observat�onWhat is the focus of interest? Is it an individual or a group of people? If it is an individual, perhaps the measurement is of an individual’s change in knowledge and behaviour due to the intervention, or maybe it involves observing a change in their health status over time. Is the focus on groups of patients (eg, patients from a particular age group, with a specific disease, or submitted to a certain intervention), or their ability to access services? If the intervention’s utility or the general quality and cost-effectiveness of different care systems is the main interest, compare the quality of care between different systems; say, between primary and secondary care or between geographic locations.

Formulate and descr�be the measure’s a�msWhat is your aim for the measure? Do you want to describe, compare or evaluate health outcomes? The selection of your instruments is highly related to the endpoints of your project. What do you want to use the instrument for?

The principal uses for a health measure are as follows.

• A health status measure can be used as an indicator, measuring the current health condition/state of a person or a population group at a point in time. In addition to validity, both reproducibility and specificity to the chosen health condition are important. Reproducibility is particularly important when undertaking a robust pre- and post-assessment of change following the intervention.

• A health outcomes measure can be used as a comparison, relating differences at different points (eg, before and after intervention). For this type of action, sensitivity and responsiveness to change are important. Put simply, the measure must be able to register small changes in people’s health.

• A health outcomes assessment implies that, apart from being an outcome measure, it is an attempt to use the information through feedback to the users of the information, including government funders of the programme. Apart from achieving the outcomes sought, government funders may also be interested in cost-effectiveness measures.

• Process outcome mon�tor�ng can use both qualitative and quantitative measures to report on process evaluation issues. This type of information helps to provide the ‘story behind the statistics’, and is useful for reporting on the following aspects that may be critical to the monitoring and success of the programme:

– community capacity

– service/agency capacity

– compliance with good practice

– programme integrity.

Examples of these types of measure are goal attainment scaling, global assessment scales and other rubric-based scoring.

1� How to Monitor for Population Health Outcomes: Guidelines for developing a monitoring framework

Cost-effect�veness measuresCost effectiveness refers to the balance between the cost of implementing the intervention and the benefit or effectiveness gained from programme. There are many ways of measuring cost effectiveness. The three approaches suggested by (The Treasury and State Services Commission 2007a; b) as being appropriate for Crown entities are:

• cost-benefit analys�s, which is seen as the ‘gold standard’ for establishing cost effectiveness

• cost per un�t of �mpact/outcome, which is a simpler approach (assuming it is feasible to quantify the impact/outcome gained)

• cost effic�ency, which is a method by which the cost per output is measured. This approach is likely to be particularly appropriate where the service provided (eg, a public health information service) is remote from any health outcome.

Cost per output is probably the most useful method when calculating the cost of a monitoring programme or a component of it. For example, the cost of purchasing information from an existing data set may be cheaper than doing it yourself. Note that cost-effectiveness is just one of a number of criteria that should be used when planning an outcomes monitoring regime.

Dec�de on the type of �nstrumentIt is important to note that the ‘psychometric’ qualities of the instrument you choose must be able to support your goals and objectives. This means it is essential that the instruments are valid and reliable (see above).

• In general, a condition-specific measure will have a narrow focus but will contain considerable detail in the area of interest. If you are interested, say, in one disease condition, and the assessment is mainly of symptoms and function, then use a condition-specific measure.

• If a specific domain, such as daily functioning or mental wellbeing in different populations, is your interest, use a dimension-specific instrument.

• If you are interested in general health or in the interaction between different conditions, or if you are interested in populations that may include healthy people, then you should use generic instruments.

• If you think the influence of other diseases or conditions that you have not measured may influence the results of the problem or the disease of interest, combine disease-specific and generic instruments.

No one instrument will prove satisfactory for all purposes. You may need to combine instruments because a reasonable instrument does not exist. But beware: when possible, use the instruments in their original form. Do not change them or use only parts of them. Validation covers only the complete instruments that were tested (see above on reliability and validity).

Be careful with instruments that are taken from one country to be used in another. Cross-cultural validation needs to follow strict rules. Even a survey that has been validated in Canada, the UK or the US may not be valid in New Zealand because of different cultural dynamics and language uses. The formal validation of an instrument is a costly and time-consuming process. How much of this work you do depends on your resources. And don’t forget the practicalities: the necessary time to fill in questionnaires and the costs of mailing and analysis. Think about your target group: for example, not every instrument suits children or older people.


The measures selected must align with the programme’s goals and objectives. It is therefore important to understand the programme’s goals and objectives so that appropriate selections can be made from the many measurement instruments available. In particular, you will need to determine whether an intervention’s effect is to be measured or descriptively assessed.

You will need broadly validated instruments that have been used in other studies if you want the intervention to describe the health status of a defined population or a specific disease category. Short, feasible and reliable instruments are recommended if care providers are to use them in their clinical work.

Collect�ng and analys�ng �nformat�onSelecting the right measures is only part of the process of establishing a monitoring system. You will also need to think about how the information will be collected and processed. There is no point choosing the perfect set of measures if there is no feasible way to collect and use the information.

Ask the following questions.

• What is the cost of collecting the required information?

• How easy is it to access the information?

• Are there administrative, privacy and ethical issues?

• Has the cost of data analysis been allowed for?

• Who is going to do the analysis?

• Who is the audience for or user of the information?

It takes time and skill to correctly analyse and interpret data, and this needs to be allowed for when planning a monitoring regime. The amount of time to allow depends on the complexity of the analysis and the type of information collected. Ask the following types of questions.

• What type of analysis is required? That is, what type of questions have to be answered: descriptive or explanatory? (Generally, explanatory analysis will require more work than descriptive analysis to prepare.)

• Who is going to use the information?

• What decisions will be made with the findings?

• When does it need to be done by?

• Who is going to do it? Do they have the skills?

In general, where the users of the information are senior decision-makers and/or the funding for the programme is significant, the more robust the analysis needs to be in order to withstand scrutiny, and consequently the more time should be allocated to this task. For major regional and national programmes involving the analysis of multiple indicators and the preparation of complex tables and report writing, a time period of several weeks or months is not unreasonable.


Part C: Issues to Cons�der �n Outcomes Mon�tor�ng

Key po�nts1. Outcomes monitoring is intended to help:

a. facilitate a process of ‘continuous programme improvement’

b. build confidence in, and support for, public health programmes.

2. Outcomes monitoring is not a cheap replacement for a traditional evaluation process.

3. Outcomes monitoring may, or may not, include a traditional formative process and impact evaluation exercise.

4. Beware of the problem of small numbers.

Outcomes mon�tor�ng as a tool for ‘cont�nuous programme �mprovement’: the problem of accountab�l�ty, attr�but�on and performance management The main motivation for introducing management tools such as programme logic models, outcomes monitoring and managing for results within the state sector is an increased expectation from central government that there will be a focus on results in the design and delivery of publicly funded services.

This expectation is highlighted by the requirements of the Public Finance Act 2004 and the Crown Entities Act 20044 (The Treasury and State Services Commission 2007a; b). This legislation means that there is now an increased emphasis on ‘maximising results for the available resources’, and managers must report on the impacts, outcomes, or objectives that a programme seeks to achieve or contribute to, and how it will contribute to implementing government policy directions (The Treasury and State Services Commission 2007a; b). Increasingly, these expectations are now being transmitted into the services contracted for by government agencies.

The reporting requirements in the acts include stating the:

• rationale for the main types of interventions planned and the results expected

• risks identified and how they will be managed

• main measures that will be used to monitor progress and performance – including the cost effectiveness of the activities

• how the programme will link to other government agency programmes to support them, including those activities undertaken by non-government organisations (The Treasury and State Services Commission 2007a; b).



Implicit in this approach is the idea of ‘accountability’ for achieving the objectives set and outcomes sought. As a general principle, the degree to which accountability can be maintained depends on the ability of the person or organisation to meaningfully influence the changes sought. Strong accountability can only be maintained where there is a clear and unambiguous causal link (ie, attribution) between cause and effect.

For many government activities, including public health programmes, establishing clear causal links between an intervention and change in the outcome sought – in this case population health – is difficult. A number of practical difficulties must be overcome (Nutbeam 1998). Consequently, when assigning attribution in public health programmes, you should beware of any ‘attributional bias’ that would result in over-attributing a change to a single programme activity.

It is likely that success will not be achieved through a single activity, but by undertaking a range of activities and working with other programmes and across-government agencies to achieve the objectives. This means that using outcomes monitoring as a strict contract performance management tool is problematic. This does not mean that non-government organisations and providers of government services should not be accountable for the choice of methodology and quality of implementation undertaken to achieve the health objectives.

The use of logic models and outcomes monitoring should be seen as a way of instituting a ‘continuous improvement cycle’ in the design and delivery of publicly funded programmes (The Treasury et al 2005), rather than as a contract performance management tool to be feared. With the continuous improvement approach, the performance of organisations is focused on their ‘understanding, reviewing, and learning from the efficiency and effectiveness of their operations’ (The Treasury et al 2005). The approach provides a tool for helping programmes to evolve over time in response to changes in the environment and as programme objectives are achieved.

This approach is a recognition that being able to attribute outcome changes to particular interventions or outputs ‘won’t always be feasible’ (The Treasury and State Services Commission 2007b). This does not absolve service providers from any responsibility and accountability for designing, implementing, monitoring and reporting on the effectiveness and efficiency of their programmes. It does mean that where establishing attribution is difficult, it is even more important that a clear logic (rationale) for the intervention be stated, and that where a number of similar but small programmes are being implemented, all the programmes use the same reporting framework and attribution takes place at the group level rather than at the individual programme provider level.

In terms of contract performance management, providers should be reassured that the approach recommended here includes considering the full range of reasons for meeting or not meeting a target. The important thing is that there is a robust programme planning and monitoring plan that will enable us to answer questions such as:

• What impact have our interventions had?

• What else was going on at the same time that also had an impact?

• Can we learn from our performance monitoring, and how can we improve in response to the lessons?


Outcomes mon�tor�ng versus evaluat�on Along with the new emphasis on using logic models for planning and monitoring for health outcomes there has been an implicit shift towards a monitoring process that relies on quantitative methods using epidemiological and biostatistical methods, rather than traditional formative and process evaluations. This shift in emphasis should not be seen as something particularly new, but rather as a reflecting the ongoing evolution in public management thinking and practice. In this case, the use of logic models can be seen as an evolving from ‘Managing by Objectives’ to ‘Managing for Outcomes’. Similarly, ‘outcomes monitoring’ reflects an evolution in the practice of ‘evaluation’.

• Traditional evaluation of public health programmes tended to focus on qualitative methods and formative and process questions, although current practice is to use multiple methods.

• Monitoring for outcomes tends to use quantitative methodologies and focuses on endpoints.

However, depending on what the programme is supposed to achieve, formative and process evaluation may still form a legitimate part of an outcome monitoring framework. For example, where community acceptance of a new type of intervention is critical to achieving the ultimate health outcome sought, then a formative and process evaluation would be appropriate.

A full programme evaluation that includes formative, process and outcomes evaluation is likely to be appropriate for major national-level campaigns. Such evaluation examines long-term changes in health status and the determinants of health. These include changes in knowledge, awareness and behaviour; shifts in social, economic and environmental conditions; as well as changes to public policy and health infrastructure.

Outcome or impact evaluation also seeks to measure the reduction in health status inequities between population subgroups. In this approach, it is important to identify and measure short-, medium- and long-term outcomes to ensure the ongoing support and relevance of the activity for those whose agendas are shorter term. Outcomes evaluation also uses indicators as benchmarks, or proxy measures, to assess the extent to which objectives have been met. Matching objectives to associated indicators in a logic model helps to ensure the availability of relevant data sources for programme evaluation.

Note that a monitoring regime using quantitative methodologies should not be seen as a cheap replacement for evaluation. Both approaches require skilled staff to advise on the most robust methods and measures to use and to undertake the data analysis, and both rely on robust administrative systems.

The problem of small numbersMonitoring the outcome of an intervention requires a health indicator that can, among other things:

• detect the outcome sought from the intervention

• be used to provide a baseline measure against which future measurements, which commence after the intervention has started, are compared

• consistently be used over the time period of interest.


The first is usually the hardest to determine, but statistical power analysis calculations and the explanations that form the logic behind the use of the indicator can help decide whether you should even begin a monitoring process using the indicator. The second is usually easy to determine, because it simply requires data before and after the intervention. The third is more difficult than it appears because it can be easy to introduce changes in the way the data is collected over time. Assessing whether the indicator has been consistently used requires someone with a good knowledge of the process.

A good health status indicator measures either:

• prevalence (the number people with the condition in the population, divided by the total number of people in the population), which can be expressed as a ratio; or

• �nc�dence (the number of new cases of the condition in the population divided by the total number of people in the population) of a disease, condition or risk factor, which can be expressed as a ratio. Generally, incidence is for a specified period of follow-up of the population of concern (eg, one year).

In both measures, the numerator or population from which one identifies the cases must match the denominator population.

The problem with these types of indicators in New Zealand is that either or both the numerator (the number of people with the health status of interest) or the denominator (the number of people in the population as a whole) may be very small. When the numbers are small, then statistically it can be difficult to measure the prevalence or incidence, and it also becomes very difficult to detect whether a change has occurred, let alone to decide whether any detectable change is related to any effect of the intervention.

Example 1: Illustrat�ng mortal�ty �nc�dence: New Zealand’s crude mortal�ty rate �n 200�In 2006, 28,390 deaths were registered in New Zealand and, as of 30 June 2006, 4,127,000 people were estimated to be living in New Zealand. So the crude mortality rate is 28,390 divided by 4,127,000, which equals 0.0069. This translates to 1 death per 145 people, or 688 deaths per 100,000 people in the denominator population.

This is an incidence rate. Mortality rates are always incidence rates due to the fact that there is no prevalent pool, because the cases are all deaths in New Zealand and so they match the same population as the denominator.

Detect�ng change �n small numbersThe ability of an indicator to detect the effect of an intervention depends on a combination of factors, including:

1. sens�t�v�ty (the ability to pick up accurately a change in the outcome measured)

2. spec�fic�ty (how accurate a test is at identifying people at risk – this is particularly important in clinical testing situations)

3. power (the probability of detecting an effect, or degree of change, of a specified size).


Determining the sensitivity and specificity of an indicator is often complicated and requires specialist knowledge beyond the scope of this How to. The power of your indicator to detect changes in the outcome of interest depends on the actual size of the effect (degree of change) taking place, or that is expected to take place based on prior experience or the literature. The larger the change, the easier it is to detect.

The power to detect also depends on the background prevalence or incidence of the indicator of interest. Finally, the size of the population of interest and the variability in the indicator go hand in hand to determine how likely it is your indicator will detect the intervention.

Following are two examples showing how the problem of small numbers in the New Zealand population inhibits our ability to statistically identify change in the health status of the population due to the effects of an intervention.

Example 2: Power to detect a change �n the number of v�olent offences �n a small commun�tyA small community has implemented a programme to reduce interpersonal violence. This community collects data on violent offences reported to the local police from one year before the programme was initiated. The community has a population of 1000 people and there are 25 violent offences reported to the police in the year before the programme. Overseas evidence from similar programmes indicates a reduction in reported offences of 15%. This translates to an annual incidence of 25/1000 or 0.025 offences per person per year. A reduction of 15% would be just under four fewer cases a year in this community.

Under this scenario, and if the reduction is actually 15%, the power of this indicator to statistically detect the reduction in the following year would be 2.7%. That means if we were to run the programme for another 37 years we would on average detect the difference once in that time. We would conclude that an indicator with higher incidence would be needed to monitor the change made due to the programme.

Figure 2 illustrates this scenario. It simulates a 10-year period for our small community: five years before and five years after the violence prevention intervention was initiated at the beginning of year six. The pattern highlights the usual statistical variation that one could expect to see. Note that in this example, because of the small numbers, it would be very hard to argue statistically that the programme has been successful – even though it might have been.


F�gure 2: S�mulated effect of 1�% reduct�on �n v�olent offences

Source: Craig Wright, Public Health Intelligence

Example 3: Power to detect the effects of a health �ntervent�onThe larger the population, or the larger the effect change that we want to detect, the easier it is to detect a change. The interaction between the size of the underlying incidence of the health issue, a desire to detect a 10% change, and population size is illustrated in Figure 3. Here, the three lines indicate the statistical power to detect a 10% change in three populations comprising 1000 people, 10,000 people, and 100,000 people, for a health problem that has an incidence of 1 to 20% in the population.

It can be seen that the population of size 1000 never has a power over 20% for the whole incidence range of 0–20%, making it unlikely to detect interventions in a population of this size. Population 10,000 reaches a reasonable level of power (ie, 80%) at about 9% incidence, so it would be unlikely that one would want to use indicators with under a 10% incidence in communities of size 10,000 when expecting a reduction of 10%. Finally, indicators in the large 100,000 size population are likely to detect a 10% decrease in the indicator above 2% incidence (with 80% confidence).

Number of offences reported

InterventionPeriod

35

30

25

20

15

10

5

01 2 3 4 5 6 7 8 9 10

Year


Power100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20%

Incidence

Population100010,000100,000

F�gure 3: Power to detect change due to an �ntervent�on, for spec�fied �nc�dence, populat�on s�ze and 10% �nc�dence reduct�on

Source: Craig Wright, Public Health Intelligence


Part D: Ava�lable Data and Informat�on from Publ�c Health Intell�gence

Key po�nts1. Public Health Intelligence has data you can use:

a. New Zealand Health Monitor Survey programme population survey data

b. Administrative data, which includes hospital records, disease registrations, and mortality data.

2. Go to PHIOnline (www.phionline.moh.govt.nz) to access data.

Public Health Intelligence (PHI) has two main types of data you can utilise as indicators for outcomes monitoring: survey data collected from population surveys, and administrative data, which includes hospital records, disease registrations, and mortality data.

Survey dataPHI is responsible for the New Zealand Health Monitor (NZHM) survey programme, which is a population-based, integrated, ongoing survey programme. Data is collected for the following three major health information domains:

• health outcomes (health status, disease states)

• causes of these outcomes (social and environmental determinants, risk and protective factors)

• health services (access, utilisation, need, coverage, quality, responsiveness, cost).

There are two main types of vehicles for collecting information in the NZHM programme: serial cross-sectional surveys and serial cohort studies. Table 5 presents a summary of the NZHM surveys.


Table �: Summary of New Zealand Health Mon�tor surveys, 2002–2012

Survey Top�c / data areasFrame (target populat�on)

Sample Mode Frequency

New Zealand Health Survey

Chronic diseases, biological and behavioural risk factors, reported health status, health service utilisation, sociodemographics.

All New Zealanders

12,500 adults (15+ years) and 5000 children (birth to 14 years)

Face-to-face, computer-assisted (CAPI) questionnaire plus anthropometric measurements in respondent’s home.

Previously 1992/93, 1996/97, and 2002/03. Now every three years (2006/07, 2009/10 etc).

Te Rau Hinengaro New Zealand Mental Health Survey

Prevalence, severity, impairment and treatment of major mental health disorders.

New Zealand adults (16+ years)

Approx. 13,000

Face-to-face CAPI questionnaire in respondent’s home.

Approximately every 10 years. Previously 2004, next planned for 2014.

New Zealand Nutrition Surveys

Food and nutrient intake, factors influencing dietary intake, nutritional status and nutrition-related status.

New Zealand adults (15+ years) or New Zealand children (5–14 years)

Approx. 4000– 5000

24-hour dietary recall and food frequency questionnaire, self-administered questionnaire, plus examination, in respondent’s home or at school for children.

Every five years, alternating between adult and child (adult 1997 next 2007/08; child 2002, next 2012).

New Zealand Oral Health Survey

Oral health status, oral health beliefs, attitudes, knowledge and practices.

All New Zealanders

Approx. 6000 to 8000

Face-to-face CAPI questionnaire and oral examination.

Every 10 years from 2008.

New Zealand Tobacco Use Survey

Tobacco use and the psychosocial correlates of smoking behaviours. Prevalence and consumption data available from the NZ Health Survey in 3rd year.

New Zealand adults (15 to 64 years)

Approx. 4000 to 6000

Face-to-face CAPI questionnaire in respondent’s home.

Two out of every three years (2005, 2006, 2008, 2009, etc).

New Zealand Alcohol and Drug Use Survey

Alcohol and illicit drug use, and the behaviours associated with alcohol and drug use.

New Zealand adults (16 to 64 years)

Approx. 8000

Face-to-face CAPI questionnaire with audio-assisted self-complete section (A-CASI) in respondent’s home.

Every two years from 2007.

Source: Ministry of Health 2005.


In addition to the cross-sectional surveys listed above, PHI is also involved in serial cohort studies (often referred to as record linkage studies). These include the New Zealand Census – Mortality Study and Cancer Trends.

Descriptive reports on the results of the above surveys can be found on PHI’s website. Access to the unit record survey data sets is available on application; see: www.moh.govt.nz/phi/surveys.

Adm�n�strat�ve dataPHI often utilises administrative data collected by the New Zealand Health Information Service (NZHIS). These data sources include hospital discharges, cancer registrations and mortality data. Table 6 describes these data sources in more detail.

Table �: Summary of adm�n�strat�ve data sources

Short name of data source

Descr�pt�on/keywordsSource of data

Per�od covered Delay

Mortality data Mortality data from the Mortality Data Collection

NZHIS 1970–2003 3 years

Foetal mortality Foetal and infant mortality data from the Mortality Data Collection


Cancer registrations All cancers NZHIS 1950–2004 3 years

Priority sites (lung, female breast, cervix, prostate and colorectal)


Notifiable diseases Notifiable diseases from Environmental Science and Research’s (ESR’s) schedule

ESR 1997–2006 18 months

Sexually transmitted infections

Sexually transmitted infections from sexual health clinics

ESR 1997–2003 18 months

Public hospital discharges

Public hospital discharge from the National Minimum Data Set (NMDS)

NZHIS via DHBSPF

1971–2006 6 months

Filtered public hospital discharges

Filtered public hospital discharge from the NMDS

NZHIS via DHBSPF

1988–2006 6 months

Private hospital discharges

Private hospital discharge from the NMDS

NZHIS 1980–1995, 2001–2003

Irregularly

Birth registrations Live and still birth registrations from the BDM registrations

BDM via SNZ

1980–2006 18 months

Mental health services Mental health services from the Mental Health Information National Collection (MHINC)

NZHIS 2001–2006 6 months

Notes: DHBSPF = District Health Board Service Provision Framework; BDM = Births, Deaths and Marriages; SNZ = Statistics New Zealand.


All of the above data sources are:

• in SAS version 8 file format

• at unit record level

• licensed to Public Health Intelligence, Ministry of Health.

PHIOnl�ne – www.ph�onl�ne.moh.govt.nzNational administrative data and survey data can often be disaggregated to the regional level, which may be more useful for your monitoring purposes. The main access portal for this information is PHIOnline. It is a powerful visualisation tool and provides an alternative way to access health information through a mapping interface rather than traditional text-based documents. Data is displayed for different DHB as well as at the Territorial Local Authority level, allowing you to visualise variability within your DHB.

F�gure 4: The PHIOnl�ne home page

The home page provides information for users on accessing the visualisation tool. Each visualisation is thought of as an ‘atlas’, and the information has been grouped in accordance to the health priorities found in the New Zealand Health Strategy. To view the atlases, just click one of the links listed on the left-hand side of the home page shown in Figure 4.

PHIOnline incorporates a number of formats to view health information on the one web page. The interface consists of an interactive map (or maps) with linked tables, charts and graphs. The standard view of the website shows the one map as the centrepiece of the interface (Figure 5).


The map is thematically drawn in relation to the range and classification of the particular data set.

PHIOnline offers a number of classifications, which the user can experiment with. To complement the map, a table of the data is also displayed, as well as a time series of the data set. You can rank the data in the table by value, highest-to-lowest or lowest-to-highest, as well as by alphabetical order. Data is displayed by region as well as providing a national figure. You can download all data displayed on the site. The site also contains extensive metadata for each data set.

F�gure �: PHIOnl�ne s�ngle map �nterface

Another interface allows you to compare two different health data sets, displaying two maps on the one web page (see Figure 6). A scatter plot for both data sets is also displayed.

In addition to survey and administrative data, PHIOnline contains data on:

• oral health

• Get Checked diabetes

• water quality

• elected services

• problem gambling.


F�gure �: PHIOnl�ne double map �nterface


GlossaryTe

rmD

efin�

t�on

Expl

anat

ory

com

men

t and

exa

mpl

e

Attr

�but

�on

Att

ribu

tion

refe

rs to

the

exte

nt to

whi

ch c

hang

e in

the

outc

ome

of

inte

rest

is a

ssoc

iate

d w

ith

the

type

of a

ctiv

ity

unde

rtak

en. S

tron

g at

trib

utio

n re

quir

es b

eing

abl

e to

est

ablis

h a

clea

r and

una

mbi

guou

s ca

usal

link

bet

wee

n ca

use

and

effe

ct. T

his

is o

ften

diffi

cult

for m

any

publ

ic h

ealt

h pr

ogra

mm

es.

Test

ing

for a

ttri

buti

on in

volv

es id

enti

fyin

g a

caus

al th

eory

and

th

en c

onfir

min

g th

e ca

usal

mec

hani

sm, u

sing

at l

east

thre

e of

the

follo

win

g w

ays.

Che

ck:

• w

ith

othe

r obs

erve

rs –

did

they

not

ice

the

sam

e th

ings

for t

he

sam

e hy

poth

esis

ed re

ason

s?

• th

at p

rogr

amm

e re

sult

s re

late

to th

e pr

ogra

mm

e co

nten

t

• th

at c

lient

impa

ct (r

esul

ts) o

ccur

red

clos

e in

tim

e to

del

iver

y of

th

e ac

tivi

ty

• th

at th

ere

are

no o

ther

suf

ficie

ntly

pla

usib

le e

xpla

nati

ons

• fo

r a p

atte

rn o

f tel

ltale

out

com

es s

ugge

stin

g on

e ca

use

or

anot

her

• th

at in

crea

sed

dose

(mor

e ac

tivi

ty) l

eads

to in

crea

sed

effe

ct (o

r gr

eate

r res

ults

)

• by

usi

ng a

com

paris

on g

roup

.

(For

mor

e, s

ee D

avid

son

2005

: 70–

84.)

Att

ribu

tion

is a

n im

port

ant i

ssue

to c

onsi

der b

ecau

se it

has

im

plic

atio

ns fo

r how

wel

l a p

rogr

amm

e’s

acti

viti

es c

an b

e sa

id

to h

ave

resu

lted

in a

ny c

hang

es o

bser

ved,

and

for a

ssig

ning

the

leve

l of a

ccou

ntab

ility

for t

he s

ucce

ss o

r fai

lure

of a

pro

gram

me

or

its

com

pone

nts.

Whe

n as

sign

ing

attr

ibut

ion,

bew

are

of a

ny ‘a

ttri

buti

onal

bia

s’ th

at

wou

ld re

sult

in o

ver-

attr

ibut

ing

a ch

ange

to a

sin

gle

prog

ram

me

acti

vity

.

For m

any

publ

ic h

ealt

h pr

ogra

mm

es, e

stab

lishi

ng c

lear

cau

sal

links

bet

wee

n an

inte

rven

tion

and

cha

nge

in a

hea

lth

outc

ome

is

diffi

cult

, and

ther

e ar

e a

num

ber o

f pra

ctic

al d

ifficu

ltie

s th

at m

ust

be o

verc

ome.

Whe

re e

stab

lishi

ng a

ttri

buti

on is

pro

blem

atic

, it

is im

port

ant t

hat a

cle

ar lo

gic

(rat

iona

le) f

or th

e in

terv

enti

on

be s

tate

d, a

nd th

at w

here

a n

umbe

r of s

imila

r pro

gram

mes

are

be

ing

impl

emen

ted

all t

he p

rogr

amm

es u

se th

e sa

me

repo

rtin

g fr

amew

ork,

and

att

ribu

tion

take

s pl

ace

at th

e gr

oup

leve

l rat

her

than

at t

he in

divi

dual

pro

gram

me

prov

ider

leve

l.

Exam

ple

1. I

f a s

ervi

ce p

rovi

der i

s co

ntra

cted

to p

rovi

de a

n in

form

atio

n se

rvic

e, th

en in

form

atio

n on

out

puts

(eg,

num

ber o

f mat

eria

ls

dist

ribu

ted,

num

ber o

f enq

uiri

es re

spon

ded

to in

a ti

me

fram

e,

leng

th o

f tim

e to

resp

ond

to re

ques

ts) i

s ap

prop

riat

e. T

his

sort

of

info

rmat

ion

will

hel

p to

pro

vide

cos

t effi

cien

cy in

form

atio

n th

at w

ill g

ive

info

rmat

ion

abou

t the

leve

l of p

erfo

rman

ce

prov

ided

by

the

serv

ice.

How

ever

, the

just

ifica

tion

for t

his

type

of i

nter

vent

ion

relie

s on

the

stre

ngth

of t

he lo

gic

for t

he

supp

ly o

f the

ser

vice

.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

2. I

f a s

ervi

ce p

rovi

der i

s co

ntra

cted

to u

nder

take

wor

kfor

ce

deve

lopm

ent,

a tr

aini

ng e

valu

atio

n m

ay in

clud

e fo

llow

-up

surv

eys

to fi

nd o

ut th

e ex

tent

to w

hich

trai

ning

con

tent

has

be

en u

sed

to re

spon

d to

and

/or r

esol

ve a

clie

nt’s

nee

ds.

3. W

here

cha

nge

in th

e he

alth

out

com

e of

inte

rest

is h

ard

to

iden

tify

(eg,

sui

cide

pre

vent

ion

in a

sm

all c

omm

unit

y), t

hen

the

effe

ctiv

enes

s of

the

inte

rven

tion

mig

ht b

e as

sess

ed b

y gr

oupi

ng s

tand

ardi

sed

data

from

all

the

prog

ram

me

prov

ider

s in

to o

ne c

olle

ctiv

e re

sult

for t

he ty

pe o

f int

erve

ntio

n as

a

who

le.

Acco

unta

b�l�t

yA

ccou

ntab

ility

refe

rs to

hol

ding

a p

erso

n or

an

orga

nisa

tion

re

spon

sibl

e fo

r obt

aini

ng th

e ob

ject

ive

set t

hrou

gh im

plem

enta

tion

of

the

inte

rven

tion

.

Bala

nced

sco

reca

rdTh

e ba

lanc

ed s

core

car

d m

odel

exp

licit

ly li

nks

orga

nisa

tion

al

stra

tegy

, org

anis

atio

nal c

apac

ity,

ser

vice

del

iver

y, s

ervi

ce

deve

lopm

ent a

nd c

lient

out

com

es. I

t is

char

acte

rised

by

feed

back

lo

ops

usin

g le

ad (m

easu

ring

eff

ort)

and

lag

(mea

suri

ng re

sult

s)

indi

cato

rs.

Caus

al o

r ex

plan

ator

y th

eory

Goo

d pr

ogra

mm

e im

plem

enta

tion

relie

s on

a fe

w c

riti

cal p

rinc

iple

s,

fore

mos

t of w

hich

is w

ell-f

ound

ed a

nd c

lear

ly s

tate

d ca

usal

(o

r exp

lana

tory

) the

ory.

Thi

s in

volv

es d

efini

ng a

cor

e pr

oble

m,

iden

tify

ing

fact

ors

that

are

pro

ven

to (o

r pot

entia

lly) c

ontr

ibut

e to

th

e pr

oble

m, a

nd id

enti

fyin

g th

e do

wns

trea

m e

ffec

ts th

is p

robl

em

crea

tes.

Cle

ar a

nd re

leva

nt p

rogr

amm

e ob

ject

ives

(see

Obj

ect�v

es)

seek

to c

hang

e ke

y co

ntri

buti

ng fa

ctor

s. S

uita

ble

indi

cato

rs (s

ee

Ind�

cato

rs) c

an th

en b

e de

velo

ped

that

che

ck w

heth

er a

ny c

hang

es

a pr

ogra

mm

e m

akes

to c

ontr

ibut

ing

fact

ors

actu

ally

pro

duce

a

redu

ctio

n in

pro

blem

eff

ects

.

In th

e fa

mily

vio

lenc

e ar

ea o

ne p

robl

em m

ight

be

defin

ed a

s ch

ildre

n’s

expo

sure

to p

aren

tal f

amily

vio

lenc

e. C

ontr

ibut

ing

fact

ors

can

rang

e ac

ross

:

• lim

ited

care

-giv

er a

war

enes

s ab

out t

he n

egat

ive

shor

t- a

nd

long

-ter

m e

ffec

ts o

f fam

ily v

iole

nce

on c

hild

ren

• ge

nera

l vol

atili

ty in

one

or b

oth

pare

nts

• in

suffi

cien

t rel

atio

nshi

p an

d pr

oble

m-s

olvi

ng s

kills

wit

h on

e or

bo

th c

areg

iver

s


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

• su

bsta

nce

abus

e an

d/or

men

tal i

llnes

s

• en

viro

nmen

tal s

tres

sors

suc

h as

une

mpl

oym

ent/

low

inco

me,

ov

ercr

owde

d ho

usin

g, s

ocia

l iso

lati

on, a

larg

er fa

mily

, ch

ildre

n’s

beha

viou

r pro

blem

s, la

ck o

f soc

ial s

uppo

rt s

ervi

ces

Prob

lem

eff

ects

may

incl

ude:

• ch

ildre

n’s

safe

ty (p

hysi

cal i

njur

y, m

enta

l hea

lth)

• ch

ildre

n’s

beha

viou

r (fr

etti

ng, a

ctin

g ou

t, a

ggre

ssio

n)

• ch

ildre

n’s

deve

lopm

ent (

enga

gem

ent w

ith

scho

ol, p

osit

ive

soci

alis

atio

n).

Usi

ng a

‘pro

blem

tree

’ app

roac

h al

low

s ap

prop

riat

e so

luti

ons

to

be id

enti

fied

and

a th

eory

of c

hang

e or

act

ion

to b

e fo

rmul

ated

(s

ee T

heor

y of

cha

nge/

act�o

n).

For m

ore,

see

:

Prob

lem

Tre

e An

alys

is (O

DI)

htt

p://

tiny

url.c

om/2

eg9g

j

Prob

lem

, Alte

rnat

ives

, Obj

ectiv

es T

ree

(Par

tici

pati

on)

http

://t

inyu

rl.c

om/2

f551

s

Cost

-eff

ect�v

enes

s

Cost

-ben

efit

Cost

per

un�

t of

�mpa

ct/o

utco

me/

outp

utCo

st-e

ffic�

ency

Cost

eff

ecti

vene

ss is

the

bala

nce

betw

een

the

cost

of d

evel

opin

g a

mea

sure

or i

ntro

duci

ng a

new

mon

itori

ng re

gim

e, a

nd th

e be

nefit

th

at w

ill b

e ga

ined

from

impl

emen

ting

it.

Ther

e ar

e m

any

way

s of

m

easu

ring

cos

t eff

ecti

vene

ss. T

he th

ree

appr

oach

es s

ugge

sted

by

(The

Tre

asur

y an

d S

tate

Ser

vice

s Co

mm

issi

on 2

007a

; b) a

s ap

prop

riat

e fo

r Cro

wn

agen

cies

and

ent

itie

s ar

e:

• co

st-b

enefi

t ana

lysi

s, w

hich

is s

een

as th

e ‘g

old

stan

dard

’ for

es

tabl

ishi

ng c

ost e

ffec

tive

ness

• co

st p

er u

nit o

f im

pact

/out

com

e, w

hich

is a

sim

pler

app

roac

h to

use

(ass

umin

g it

is fe

asib

le to

qua

ntif

y th

e im

pact

/ o

utco

me

gain

ed)

For m

any

prog

ram

mes

, a ju

dgem

ent w

ill h

ave

to b

e m

ade

as to

w

heth

er th

e co

st o

f dev

elop

ing

a m

onito

ring

regi

me

or m

easu

re

is w

orth

the

bene

fit to

be

gain

ed fr

om th

e us

e of

the

info

rmat

ion

prov

ided

. Whe

re th

e co

st o

utw

eigh

s th

e be

nefit

, con

side

r usi

ng

an a

ppro

pria

te p

roxy

indi

cato

r, o

r whe

ther

info

rmat

ion

from

an

alre

ady

exis

ting

mon

itori

ng re

gim

e co

uld

be u

sed.

In g

ener

al, t

he p

ropo

rtio

n of

the

tota

l bud

get s

pent

on

mon

itori

ng

shou

ld b

e in

the

regi

on o

f 5%

to 2

0%, d

epen

ding

on

the

type

and

si

ze o

f the

pro

gram

me.

Whe

re th

e pr

ogra

mm

e is

a p

ilot i

nitia

tive

, th

e re

sult

s of

whi

ch c

ould

be

used

to s

igni

fican

tly

influ

ence

w

heth

er th

e pr

ogra

mm

e is

exp

ande

d in

to a

maj

or e

ffor

t, th

en

20%

of t

he to

tal b

udge

t cou

ld b

e ap

prop

riat

e.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

• co

st-e

ffici

ency

, whi

ch is

a m

etho

d by

whi

ch th

e co

st p

er o

utpu

t is

mea

sure

d. T

his

appr

oach

is li

kely

to b

e pa

rtic

ular

ly a

ppro

pria

te

whe

re th

e se

rvic

e pr

ovid

ed (e

g, a

pub

lic h

ealt

h in

form

atio

n se

rvic

e) is

rem

ote

from

any

hea

lth

outc

ome.

How

ever

, if t

he p

rogr

amm

e ha

s a

larg

e bu

dget

and

the

prog

ram

me

is w

ell e

stab

lishe

d, a

pplie

d in

sta

ndar

d w

ays

and

supp

orte

d by

evi

denc

e fo

r its

eff

ecti

vene

ss, t

hen

a bu

dget

al

loca

tion

of 5

% m

ay b

e ap

prop

riat

e fo

r mon

itori

ng p

urpo

ses.

Cost

per

out

put i

s pr

obab

ly th

e be

st m

etho

d to

use

whe

n ca

lcul

atin

g th

e co

st o

f a m

onito

ring

pro

gram

me

or a

com

pone

nt

of it

. Fo

r exa

mpl

e, th

e co

st o

f pur

chas

ing

info

rmat

ion

from

an

exis

ting

dat

a se

t may

be

chea

per t

han

doin

g it

your

self.

Or,

it m

ay

be c

heap

er to

use

a p

roxy

mea

sure

than

to u

se a

dir

ect m

easu

re

– a

ssum

ing

the

prox

y m

easu

re is

robu

st e

noug

h to

wit

hsta

nd

publ

ic s

crut

iny

and

the

info

rmat

ion

trad

e-of

f is

acce

ptab

le. N

ote

that

cos

t eff

ecti

vene

ss is

just

one

of a

num

ber o

f cri

teri

a th

at

shou

ld b

e us

ed in

pla

nnin

g an

out

com

es m

onito

ring

regi

me.

Dat

a

D

ata

sets

Dat

a is

fact

ual i

nfor

mat

ion

that

aft

er a

naly

sis

is u

sed

to re

ason

or

mak

e de

cisi

ons.

The

wor

d is

com

mon

ly u

sed

to re

fer t

o ei

ther

sin

gle

or m

ulti

ple

piec

es o

f inf

orm

atio

n, a

nd in

tech

nica

l rep

orts

is o

ften

us

ed in

the

plur

al fo

rm (a

s in

‘dat

a ar

e co

llect

ed’)

.

Dat

a se

ts re

fers

to c

olle

ctio

ns o

f pie

ces

of in

form

atio

n th

at a

re

colle

cted

for p

arti

cula

r pur

pose

s. ‘D

ata

sets

’ is

syno

nym

ous

wit

h da

taba

ses.

For e

xam

ple,

the

Phar

mho

use

data

set

col

lect

s pi

eces

of

info

rmat

ion

abou

t the

pre

scri

bing

of d

rugs

sub

sidi

sed

in N

ew

Zeal

and

by P

harm

ac.

The

New

Zea

land

Hea

lth

Info

rmat

ion

Ser

vice

is re

spon

sibl

e fo

r the

co

llect

ion

and

mai

nten

ance

of m

any

of th

e la

rge

nati

onal

hea

lth-

rela

ted

data

set

s, s

uch

as m

orta

lity

and

mor

bidi

ty d

ata.

Oth

er

gove

rnm

ent a

genc

ies

are

resp

onsi

ble

for o

ther

dat

a se

ts th

at

can

be u

sefu

l for

mon

itori

ng p

ublic

hea

lth

prog

ram

mes

. A la

rge

num

ber o

f the

se d

ata

sets

are

hos

ted

on th

e PH

IOnl

ine

web

site

(w

ww

.phi

onlin

e.m

oh.g

ovt.

nz).


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Eval

uat�o

n

F

orm

at�v

e

P

roce

ss

I

mpa

ct

Eval

uati

on is

a re

sear

ch p

roce

ss th

at a

ims

to a

sses

s sy

stem

atic

ally

an

d ob

ject

ivel

y th

e re

leva

nce,

per

form

ance

and

suc

cess

of o

ngoi

ng

or c

ompl

eted

pro

gram

mes

. Ev

alua

tion

aim

s to

pro

vide

info

rmat

ion

abou

t whe

ther

und

erly

ing

assu

mpt

ions

or t

heor

ies

used

in th

e pr

ogra

mm

e de

velo

pmen

t wer

e va

lid, w

hat w

orke

d an

d w

hat d

id n

ot

wor

k, a

nd w

hy. T

ypic

ally

eva

luat

ion

aim

s to

det

erm

ine

the

rele

vanc

e,

effic

ienc

y, e

ffec

tive

ness

, im

pact

and

sus

tain

abili

ty o

f a p

rogr

amm

e.

Eval

uati

on s

houl

d al

low

for l

esso

ns le

arnt

to b

e tr

ansf

erre

d in

to

othe

r pro

gram

mes

.

A c

ompr

ehen

sive

eva

luat

ion

prog

ram

me

usua

lly in

volv

es th

ree

part

s:

• fo

rmat

ive

eval

uati

on

• pr

oces

s ev

alua

tion

• im

pact

/out

com

e ev

alua

tion

.

Form

ativ

e ev

alua

tion

focu

ses

on th

e es

tabl

ishm

ent o

f the

pr

ogra

mm

e. I

t is

part

icul

arly

use

ful w

hen

the

prog

ram

me

is

new

, unt

este

d an

d re

quir

es c

omm

unit

y ac

cept

ance

if it

is to

be

suc

cess

ful.

It se

eks

to u

nder

stan

d w

hat t

he c

ondi

tion

s fo

r pr

ogra

mm

e su

cces

s ar

e su

ppos

ed to

be.

Proc

ess

eval

uatio

n fo

cuse

s on

how

wel

l the

pro

gram

me

func

tion

s.

Are

the

cond

itio

ns fo

r suc

cess

pre

sent

in th

e w

ay th

e pr

ogra

mm

e is

be

ing

deliv

ered

?

Impa

ct /

outc

ome

eval

uatio

n as

sess

es h

ow s

ucce

ssfu

l the

pr

ogra

mm

e ha

s be

en in

del

iver

ing

the

desi

red

outc

omes

. Im

pact

ev

alua

tion

sho

uld

alw

ays

try

to id

enti

fy u

nint

ende

d be

nefit

s an

d an

y ne

gati

ve p

rogr

amm

e co

nseq

uenc

es.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Oth

er te

rms

that

may

be

asso

ciat

ed w

ith

eval

uati

on in

clud

e:

• ou

tput

• ou

tcom

e

• im

pact

.

Ev�d

ence

d-ba

sed

publ

�c h

ealth

(E

BPH

)

Defi

niti

ons

of w

hat ‘

evid

ence

d ba

sed

publ

ic h

ealt

h’ m

eans

are

ev

olvi

ng (K

ohat

su e

t al 2

004)

. In

the

cont

ext o

f the

Gui

de to

D

evel

opin

g Pu

blic

Hea

lth P

rogr

amm

es: A

gen

eric

pro

gram

me

logi

c m

odel

(Min

istr

y of

Hea

lth

2006

) and

this

How

to, t

he d

efini

tion

by

Bro

wns

on (2

003)

is p

refe

rred

:

EB

PH is

the

deve

lopm

ent,

impl

emen

tati

on, a

nd e

valu

atio

n of

eff

ecti

ve p

rogr

ams

and

polic

ies

in p

ublic

hea

lth

thro

ugh

appl

icat

ion

of p

rinc

iple

s of

sci

enti

fic re

ason

ing,

incl

udin

g sy

stem

atic

use

s of

dat

a an

d in

form

atio

n sy

stem

s an

d ap

prop

riat

e us

e of

pro

gram

pla

nnin

g m

odel

s.

See

als

o Ko

hats

u (2

004)

and

Jeni

cek

(199

7) fo

r alte

rnat

ive

form

ulat

ions

defi

ning

EB

PH. K

ohat

su’s

(200

4) fo

rmul

atio

n is

ap

plic

able

to th

e po

licy

deve

lopm

ent p

roce

ss, w

hile

Jeni

cek’

s (1

997)

defi

niti

on is

an

appl

icat

ion

of e

vide

nce-

base

d m

edic

ine

conc

epts

to p

ublic

hea

lth

prac

tice

.

(See

als

o Pu

blic

hea

lth.

)

Impa

cts

Impa

cts

are

the

cont

ribu

tion

mad

e to

the

achi

evem

ent o

f an

outc

ome

by a

spe

cifie

d se

t of a

ctio

ns (T

he T

reas

ury

and

Sta

te

Ser

vice

s Co

mm

issi

on 2

007b

).

Impa

cts

shou

ld b

e po

siti

ve, a

ltho

ugh

unin

tend

ed c

onse

quen

ces

from

an

acti

on m

ay o

ccur

that

resu

lt in

eit

her p

osit

ive

or n

egat

ive

cont

ribu

tion

s to

the

achi

evem

ent o

f the

des

ired

out

com

e.

Not

e: th

is d

efini

tion

and

use

of ‘

impa

ct’ a

re d

iffer

ent t

o ho

w

it is

defi

ned

and

used

in e

valu

atio

n m

etho

dolo

gy (s

ee a

lso

Eval

uati

on).


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Ind�

cato

rs

Q

uant

�tat�v

e

Q

ual�t

at�v

e

Pr

oxy

Indi

cato

rs a

re e

ithe

r qua

ntita

tive

or q

ualit

ativ

e m

easu

res

that

as

sess

the

dire

ctio

n an

d si

ze o

f cha

nge

in th

e th

ing

bein

g m

easu

red.

Qua

ntita

tive

indi

cato

rs a

re n

umer

ical

mea

sure

men

ts th

at le

nd

them

selv

es to

sta

tisti

cal a

naly

sis.

(Thi

s H

ow to

focu

ses

on th

is

type

of i

ndic

ator

). Q

ualit

ativ

e in

dica

tors

are

pri

mar

ily m

easu

res

take

n in

a te

xtua

l (na

rrat

ive)

form

(ie,

non

-num

eric

al) a

nd m

ay o

r m

ay n

ot le

nd th

emse

lves

to q

uant

ifica

tion

. Ana

lysi

s of

qua

litat

ive

indi

cato

rs re

quir

es th

e us

e of

qua

litat

ive

anal

ytic

al te

chni

ques

. Pr

oxy

indi

cato

rs a

re m

easu

res

that

pro

vide

an

appr

oxim

ate

esti

mat

e of

cha

nge

in th

e ou

tcom

e of

inte

rest

. Pro

xy m

easu

res

may

be

appr

opri

ate

to u

se w

hen

dire

ct m

easu

res

cann

ot b

e us

ed b

ecau

se

of c

once

rns

abou

t eth

ics,

cos

t, c

ompl

exit

y of

mea

sure

men

t or

tim

elin

ess.

For

exa

mpl

e, h

ospi

talis

atio

ns fo

r int

enti

onal

sel

f-ha

rm

are

used

as

an a

ccep

tabl

e pr

oxy

mea

sure

of s

uici

de a

ttem

pt, a

nd

BM

I is

usua

lly a

ccep

ted

as a

pro

xy fo

r obe

sity

.

(See

als

o In

stru

men

t.)

Tim

e is

an

impo

rtan

t fac

tor i

n as

sess

ing

wha

t cha

nge

has

occu

rred

. In

dica

tors

may

mea

sure

cha

nge

over

the

shor

t,

med

ium

or l

ong

term

. Wha

t con

stitu

tes

shor

t, m

ediu

m o

r lon

g te

rm d

epen

ds o

n th

e co

ntex

t and

the

issu

e(s)

the

prog

ram

me

is

tack

ling.

Hea

lth

indi

cato

rs d

irec

tly

or in

dire

ctly

(ie,

pro

xy) m

easu

re a

he

alth

-rel

ated

cha

ract

eris

tic

of a

n in

divi

dual

, pop

ulat

ion

or th

e en

viro

nmen

t. T

he in

dica

tor m

ay m

easu

re o

ne o

r mor

e he

alth

as

pect

s (q

ualit

y, q

uant

ity

and

tim

e) (N

utbe

am 1

998)

.

Inpu

tsRe

sour

ces

put i

nto

a pr

ogra

mm

e to

car

ry o

ut a

n ac

tivi

ty. A

noth

er

term

for i

nput

is ‘e

ffor

t’.

Inpu

ts c

an b

e hu

man

, mat

eria

l, fin

anci

al o

r exp

ress

ed a

s ti

me.

Ef

fort

des

crib

es e

very

thin

g th

at ‘w

e’ u

se a

nd d

o fo

r ‘th

em’ –

the

prog

ram

me

reci

pien

ts.

Inst

rum

ent

An

inst

rum

ent i

s a

num

eric

al o

r tex

t-ba

sed

mea

suri

ng m

etho

d th

at

show

s th

e ex

tent

or a

mou

nt o

r qua

ntit

y or

deg

ree

of s

omet

hing

of

inte

rest

.

The

inst

rum

ent i

s th

e pr

oces

s us

ed to

pro

duce

the

desi

red

publ

ic

heal

th in

dica

tor.

(See

Ind�

cato

rs.)

Mea

sure

(s)

See

Ind�

cato

rs a

nd In

stru

men

t.

Met

r�c(

s)S

ee In

d�ca

tors

and

Inst

rum

ent.

Mon

�tor

See

Out

com

es m

on�to

r�ng

.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Obj

ect�v

es H

�gh-

orde

r/ge

nera

l S

pec�

fic/

�nt

erm

ed�a

te L

ow-o

rder

/

ope

rat�o

nal

SM

ART

Obj

ecti

ves

are

stat

emen

ts a

bout

the

resu

lts

a pr

ogra

mm

e se

eks

to a

chie

ve. A

ny p

rogr

amm

e m

ust h

ave

at le

ast o

ne o

bjec

tive

. O

bjec

tive

s m

ay b

e tr

ansl

ated

dir

ectl

y in

to ‘o

utco

mes

’ whe

re th

ey

deal

wit

h on

ly o

ne is

sue.

How

ever

, dou

ble-

barr

elle

d ob

ject

ives

will

re

quir

e m

ulti

ple

outc

ome

mea

sure

s to

be

deve

lope

d.

Obj

ecti

ves

may

form

a h

iera

rchy

that

mov

es fr

om a

lim

ited

set o

f hi

gh-o

rder

obj

ecti

ves

that

are

syn

onym

ous

wit

h ai

ms

or g

oals

, to

low

er-o

rder

or m

ore

spec

ific

or in

term

edia

te o

bjec

tive

s, a

s fo

llow

s.

Hig

h-or

der o

r gen

eral

obj

ectiv

es a

re s

tate

men

ts a

bout

the

over

-all

or lo

ng-t

erm

eff

ects

or u

ltim

ate

outc

omes

that

are

exp

ecte

d to

be

obta

ined

as

a re

sult

of th

e pr

ogra

mm

e, o

r to

whi

ch th

e pr

ogra

mm

e co

ntri

bute

s to

as

part

of a

who

le-o

f-go

vern

men

t ini

tiati

ve.

Att

ainm

ent o

f hig

h-or

der o

bjec

tive

s m

ay o

r may

not

be

dire

ctly

at

trib

utab

le to

any

one

pro

gram

me.

Lon

g-te

rm o

bjec

tive

s ar

e ge

nera

lly th

ose

to b

e ac

hiev

ed in

5 to

10

year

s.

Spec

ific

or in

term

edia

te-l

evel

obj

ectiv

es a

re s

tate

men

ts a

bout

th

e in

term

edia

te o

utco

mes

that

are

exp

ecte

d to

be

achi

eved

by

the

prog

ram

me,

and

that

are

mor

e di

rect

ly a

ttri

buta

ble

to th

e pr

ogra

mm

e’s

acti

viti

es. T

hese

obj

ecti

ves

are

gene

rally

thos

e to

be

achi

eved

in 3

to 5

yea

rs.

Low

-ord

er o

r ope

ratio

nal o

bjec

tives

are

sta

tem

ents

abo

ut th

e im

med

iate

out

com

es (w

hich

may

be

expr

esse

d as

out

puts

in

som

e ci

rcum

stan

ces)

exp

ecte

d to

be

achi

eved

or a

ccom

plis

hed

from

the

inte

rven

tion

that

are

dir

ectl

y at

trib

utab

le to

the

acti

viti

es

unde

rtak

en. T

hese

obj

ecti

ves

are

gene

rally

thos

e to

be

achi

eved

in

1 to

2 y

ears

.

In te

rms

of th

e Pu

blic

Fin

ance

Act

200

4, o

bjec

tive

s ar

e no

t ex

pres

sly

defin

ed, a

nd c

onse

quen

tly

can

be s

een

as m

eani

ng a

go

al o

r aim

. How

ever

, bec

ause

som

e co

re d

epar

tmen

tal a

ctiv

itie

s th

at s

ervi

ce th

e op

erat

ion

of g

over

nmen

t do

not fi

t wit

hin

the

defin

itio

n of

an

outc

ome

give

n in

the

Act

, obj

ecti

ves

are

also

see

n as

bei

ng s

imila

r to

impa

cts

and

outc

omes

. (S

ee O

utco

mes

.) (T

he

Trea

sury

and

Sta

te S

ervi

ces

Com

mis

sion

200

7a.)

In h

ealt

h te

rms,

hea

lth

goal

s (o

bjec

tive

s/ai

ms)

are

gen

eral

st

atem

ents

of i

nten

t and

dir

ecti

on s

et fo

r a h

ealt

h pr

ogra

mm

e,

whi

ch m

ay in

clud

e a

heal

th ta

rget

to a

chie

ve (N

utbe

am 1

998)

. In

this

doc

umen

t, th

e te

rm ‘o

bjec

tive

’ is

used

to id

enti

fy

part

icul

ar re

sult

s th

e pr

ogra

mm

e se

eks

to a

chie

ve. F

or m

onito

ring

pu

rpos

es, t

he re

sult

may

be

expr

esse

d in

term

s of

an

outc

ome

or

an o

utpu

t. (S

ee O

utco

mes

and

Out

puts

.)

Follo

win

g is

an

exam

ple

of a

hie

rarc

hy o

f obj

ecti

ves.

Hig

h-or

der o

bjec

tive

(syn

onym

ous

wit

h vi

sion

, aim

or g

oal)

:

• En

able

peo

ple

wit

h ch

roni

c co

ndit

ions

to im

prov

e th

eir

heal

th, s

low

pro

gres

s of

thei

r con

diti

on(s

), a

nd m

aint

ain

inde

pend

ence

whe

reve

r pos

sibl

e by

alig

ning

com

mun

ity

and

hosp

ital s

ervi

ces

acro

ss [

] D

HB

.

Inte

rmed

iate

-lev

el o

bjec

tive

asso

ciat

ed w

ith a

chie

ving

the

high

-le

vel o

bjec

tive:

• Re

duce

the

inci

denc

e of

can

cer,

dia

bete

s, a

nd c

ardi

ovas

cula

r di

seas

e by

20%

resp

ecti

vely

ove

r the

nex

t 5 y

ears

.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Obj

ecti

ves

shou

ld b

e w

ritt

en a

s SM

ART

obje

ctiv

es:

• Sp

ecifi

c

• M

easu

rabl

e

• Ac

hiev

able

(som

etim

es re

ferr

ed to

as

‘acc

urat

e’ o

r ‘ac

tion

-or

ient

ed’)

• Re

leva

nt (s

omet

imes

refe

rred

to a

s ‘r

ealis

tic’

)

• Ti

me-

base

d.

Low

-ord

er o

r ope

rati

onal

obj

ecti

ves

asso

ciat

ed w

ith

achi

evin

g th

e in

term

edia

te le

vel:

• In

crea

se th

e le

vel o

f kno

wle

dge

by 5

0% in

the

targ

et

popu

lati

on g

roup

s ab

out t

he c

ause

s an

d pr

even

tion

of b

reas

t an

d pr

ostr

ate

canc

er, d

iabe

tes

and

card

iova

scul

ar d

isea

se.

• In

crea

se th

e up

take

of c

ance

r scr

eeni

ng s

ervi

ces

by 5

0% in

the

high

-ris

k po

pula

tion

s in

the

next

3 y

ears

.

• In

crea

se a

cces

s by

50%

to e

arly

dia

gnos

is s

ervi

ces

in ru

ral

com

mun

itie

s in

the

next

2 y

ears

.

Out

com

es

In

term

ed�a

te

Cr

oss-

agen

cy

Pr

oces

s

Out

com

es a

re s

peci

fic s

tate

men

ts a

bout

the

inte

nded

cha

nge

in

publ

ic h

ealt

h-re

late

d at

titu

des,

kno

wle

dge,

beh

avio

urs,

or p

hysi

cal

(inc

ludi

ng m

enta

l) h

ealt

h st

atus

in th

e ta

rget

pop

ulat

ion(

s) s

ough

t by

und

erta

king

the

plan

ned

publ

ic h

ealt

h ac

tivi

ty.

Inte

rmed

iate

out

com

es (a

lso

calle

d in

teri

m o

utco

mes

) are

ste

ps

alon

g th

e w

ay to

the

desi

red

end

outc

ome.

The

y ar

e of

ten

smal

ler

chan

ges

that

nee

d to

hap

pen

befo

re th

e fin

al d

esir

ed o

utco

me

can

be re

ache

d.

Cros

s-ag

ency

out

com

es a

re o

utco

mes

whe

re th

e co

llect

ive

effo

rt o

f m

ore

than

one

gov

ernm

ent a

genc

y or

pro

gram

me

is re

quir

ed if

the

outc

ome

is to

be

achi

eved

.

Proc

ess

outc

omes

typi

cally

mea

sure

the

amou

nt o

f eff

ort p

ut in

to a

pr

ogra

mm

e an

d th

e qu

alit

y of

the

serv

ice

prov

ided

. The

y ca

n al

so

be a

ppro

pria

te w

here

it is

impo

rtan

t to

mon

itor c

omm

unit

y su

ppor

t fo

r a p

rogr

amm

e. P

roce

ss o

utco

mes

that

mea

sure

eff

ort c

an b

e ex

pres

sed

as o

utpu

ts.

(See

Pro

cess

out

com

e m

on�to

r�ng

.)

Hea

lth

outc

omes

sho

uld

be e

xpre

ssed

in s

uch

a w

ay th

at

indi

cate

s th

e ty

pe, d

irec

tion

and

ext

ent o

f cha

nge

in h

ealt

h kn

owle

dge,

beh

avio

ur o

r sta

tus

soug

ht in

an

indi

vidu

al o

r po

pula

tion

gro

up.

Exam

ple

• Ra

tes

of s

mok

ing

cess

atio

n in

you

ng w

omen

und

er 2

0 ye

ars

of

age

incr

ease

by

20%

at t

he e

nd o

f the

pro

gram

me.

Att

ainm

ent o

f the

out

com

es is

mea

sure

d th

roug

h in

dica

tors

(see

In

d�ca

tors

). Id

eally

, the

hea

lth

outc

ome

soug

ht s

houl

d be

dir

ectl

y at

trib

utab

le to

the

inte

rven

tion

, but

this

is u

sual

ly p

robl

emat

ic in

he

alth

pro

mot

ion

acti

viti

es (N

utbe

am 1

998)

.

This

defi

niti

on c

an b

e se

en a

s a

spec

ific

appl

icat

ion

to p

ublic

he

alth

of t

he d

efini

tion

of a

n ‘o

utco

me’

pro

vide

d in

the

Publ

ic

Fina

nce

Act

200

4 (s

2(1)

): ‘A

sta

te o

r con

diti

on o

f soc

iety

, the

ec

onom

y or

the

envi

ronm

ent a

nd in

clud

es a

cha

nge

in th

at s

tate

or

con

diti

on.’


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Cros

s-ag

ency

out

com

es s

houl

d pr

obab

ly b

e th

e ru

le ra

ther

than

th

e ex

cept

ion

in c

ompr

ehen

sive

pub

lic h

ealt

h pr

ogra

mm

es. I

n su

ch s

ituat

ions

, it i

s im

port

ant t

hat t

he ro

les

and

resp

onsi

bilit

ies

of th

e pa

rtic

ipat

ing

agen

cies

are

agr

eed

– in

clud

ing

who

is

resp

onsi

ble

for m

onito

ring

out

com

es –

bef

ore

the

prog

ram

me

begi

ns.

Out

com

es

mon

�tor�

ngA

pla

n fo

r the

rout

ine,

sys

tem

atic

col

lect

ion

and

reco

rdin

g of

in

form

atio

n ab

out a

spec

ts o

f a p

rogr

amm

e ov

er ti

me.

The

pur

pose

is

to a

sses

s w

heth

er p

rogr

ess

is b

eing

mad

e on

ach

ievi

ng th

e pr

ogra

mm

e ob

ject

ives

.

Prep

arin

g th

e ou

tcom

es m

onito

ring

pla

n re

quir

es a

sses

sing

:

• w

hat n

eeds

to b

e m

easu

red

to d

emon

stra

te s

ucce

ss

• ho

w th

ings

are

goi

ng to

be

mea

sure

d an

d by

who

m (d

ecid

ing

wha

t ind

icat

ors

are

goin

g to

be

used

, aft

er c

onsi

deri

ng

issu

es s

uch

as v

alid

ity,

relia

bilit

y, s

ensi

tivi

ty, a

ttri

buti

on a

nd

feas

ibili

ty) (

see

Ind�

cato

rs.)

•

the

tim

efra

mes

for d

eliv

erin

g in

form

atio

n ab

out t

he

perf

orm

ance

of t

he p

rogr

amm

e ag

ains

t the

sta

ted

obje

ctiv

es

to d

ecis

ion-

mak

ers

and

key

stak

ehol

ders

in a

tim

ely

man

ner.

Out

com

es m

onito

ring

may

or m

ay n

ot in

clud

e a

form

al e

valu

atio

n pr

oces

s de

pend

ing

on th

e ob

ject

ives

of t

he p

rogr

amm

e. (S

ee

Eval

uat�o

n.)

Out

puts

Out

puts

are

thin

gs (e

g, g

oods

) pro

duce

d, s

ervi

ces

deliv

ered

, ev

ents

hel

d, o

r par

tici

pati

on g

ener

ated

resu

ltin

g fr

om th

e ac

tivi

ties

un

dert

aken

. (S

ee O

utco

mes

and

Pro

cess

out

com

e m

on�to

r�ng

).

This

defi

niti

on is

con

sist

ent w

ith

that

pro

vide

d in

the

Publ

ic

Fina

nce

Act

200

4. C

omm

on p

ublic

hea

lth

outp

uts

are

thin

gs s

uch

as in

form

atio

n se

rvic

es p

rovi

ded,

mee

ting

s he

ld o

r att

ende

d,

and

educ

atio

nal/

trai

ning

or s

ocia

l mar

keti

ng p

rodu

cts

or s

ervi

ces

deliv

ered

. Suc

h ou

tput

s ar

e in

tend

ed to

hav

e an

impa

ct th

at

assi

sts

or re

sult

s in

ach

ievi

ng th

e de

sire

d pu

blic

hea

lth

outc

ome,

su

ch a

s a

chan

ge in

hea

lth

know

ledg

e an

d be

havi

our.

(See

Out

com

es a

nd P

roce

ss o

utco

me

mon

�tor�

ng.)


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Proc

ess

outc

ome

mon

�tor�

ngTh

is is

like

out

com

e m

onito

ring

but

incl

udes

kee

ping

trac

k of

how

w

ell a

pro

gram

me

is b

eing

impl

emen

ted

in it

s di

ffer

ent a

spec

ts.

Proc

ess

outc

ome

mon

itori

ng p

rovi

des

the

basi

s fo

r pro

gram

me

staf

f to

ass

ure

them

selv

es th

at g

ood

outc

omes

are

ass

ocia

ted

wit

h go

od

prog

ram

me

deliv

ery.

If t

he d

esir

ed o

utco

mes

are

not

ach

ieve

d, th

en

the

agen

cy w

ill h

ave

suffi

cien

t inf

orm

atio

n to

ass

ess

whi

ch a

spec

ts

of p

rogr

amm

e de

liver

y ne

ed to

be

deve

lope

d or

cha

nged

.

(See

als

o Ba

lanc

ed s

core

card

.)

A p

rogr

amm

e th

eory

of c

hang

e w

ill in

dica

te w

hat k

inds

of i

nput

s an

d pr

oces

ses

are

likel

y to

pro

duce

a p

reve

ntio

n ef

fect

. For

ex

ampl

e, a

hom

e-vi

siti

ng p

rogr

amm

e m

ight

det

ail t

he fo

llow

ing

as v

ital t

o go

od c

lient

out

com

es:

• ap

prop

riat

ely

expe

rien

ced,

ski

lled

and

supe

rvis

ed s

taff

• st

aff w

ho a

re e

mpa

thet

ic a

nd w

arm

in th

eir i

nter

acti

ons

• a

fam

ily/w

hana

u as

sess

men

t tha

t ide

ntifi

es s

tren

gths

and

ch

alle

nges

like

ly to

hel

p or

hin

der c

hild

-rea

ring

(eg,

incl

udes

a

scre

en fo

r fam

ily v

iole

nce,

sub

stan

ce a

buse

, dep

ress

ion)

• th

e as

sess

men

t pro

cess

enc

oura

ges

co-o

pera

tion

and

redu

ces

defe

nsiv

enes

s

• fa

mili

es a

re v

isite

d at

suf

ficie

ntly

regu

lar i

nter

vals

• id

eally

the

ethn

icit

y of

wor

kers

and

pri

mar

y ca

regi

vers

are

m

atch

ed to

max

imis

e en

gage

men

t

• th

ere

is a

dev

elop

men

tal m

odel

to c

ompa

re c

hild

ren’

s pr

ogre

ss a

gain

st

• th

e de

velo

pmen

tal m

odel

is e

asy

to c

omm

unic

ate

to

care

give

rs a

nd th

ey ‘b

uy’ i

t

• de

velo

pmen

tal o

utco

mes

for c

hild

ren

and

cond

itio

ns th

at

hind

er n

orm

al d

evel

opm

ent a

re s

yste

mat

ical

ly tr

acke

d

• gu

idan

ce is

pro

vide

d in

a re

spon

sive

way

(ie,

is ti

mel

y,

solu

tion

focu

sed,

real

isti

c an

d ch

ecke

d fo

r und

erst

andi

ng)

• go

al-s

etti

ng is

use

d to

enh

ance

car

egiv

er m

otiv

atio

n an

d to

pr

iori

tise

acti

ons.

Regu

lar s

uper

visi

on m

eeti

ngs

and/

or p

rogr

amm

e re

view

s pr

ovid

e sp

ace

for c

lient

s’ p

rogr

ess

to b

e ch

ecke

d ag

ains

t the

qua

lity

of

prog

ram

me

deliv

ery.

Any

cha

nges

in c

lient

pla

ns, p

rogr

amm

e de

sign

or p

rofe

ssio

nal d

evel

opm

ent c

an b

e co

nsid

ered

a p

roce

ss

outc

ome.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Publ

�c h

ealth

Popu

lat�o

n-ba

sed

publ

�c h

ealth

Pu

bl�c

hea

lth

act�o

n

Ther

e is

a ra

nge

of te

rms

and

defin

itio

ns re

lati

ng to

pub

lic h

ealt

h,

incl

udin

g pu

blic

hea

lth

acti

on a

nd p

opul

atio

n-ba

sed

publ

ic h

ealt

h.

The

Wor

ld H

ealt

h O

rgan

izat

ion

(199

8) d

efine

s pu

blic

hea

lth

as:

Th

e sc

ienc

e an

d ar

t of p

rom

otin

g he

alth

, pre

vent

ing

dise

ase,

and

pr

olon

ging

life

thro

ugh

the

orga

nize

d ef

fort

s of

soc

iety

.

It ca

n be

arg

ued

that

this

defi

niti

on o

mit

s th

e im

port

ant p

oint

that

pu

blic

hea

lth

prim

arily

focu

ses

on th

e he

alth

of t

he p

opul

atio

n or

po

pula

tion

sub

-gro

ups

and

not t

he h

ealt

h of

spe

cific

indi

vidu

als.

Th

is fo

cus

on p

opul

atio

ns in

pub

lic h

ealt

h is

iden

tifie

d m

ore

expl

icit

ly in

mor

e re

cent

defi

niti

ons.

• ‘P

ublic

hea

lth

is c

once

rned

wit

h im

prov

ing

the

heal

th o

f the

po

pula

tion

, rat

her t

han

trea

ting

the

dise

ases

of i

ndiv

idua

l pa

tien

ts’

(Dep

artm

ent o

f Hea

lth

2004

cite

d in

Pub

lic H

ealt

h el

ectr

onic

Lib

rary

).

• ‘[P

ublic

hea

lth

is] c

once

rned

wit

h th

e he

alth

of t

he c

omm

unit

y as

a w

hole

’ (Am

eric

an Jo

urna

l of P

ublic

Hea

lth 2

006,

web

site

(h

ttp:

//w

ww

.ajp

h.or

g/))

.

• Pu

blic

hea

lth

acti

on is

‘Co

llect

ive

acti

on fo

r sus

tain

ed

popu

lati

on-w

ide

heal

th im

prov

emen

t … a

nd to

redu

ce h

ealt

h in

equa

litie

s. R

espo

nsib

ility

for s

uch

acti

on is

not

con

fined

to

the

heal

th s

ecto

r but

sho

uld

incl

ude

all s

ecto

rs w

hose

ac

tion

s af

fect

the

heal

th o

f pop

ulat

ions

. Pop

ulat

ions

can

be

geo

grap

hic

and/

or d

efine

d by

fact

ors

such

as

ethn

icit

y,

gend

er, a

ge, s

exua

l ori

enta

tion

, inc

ome

etc.

’ (B

eagl

ehol

e et

al

2004

; Pub

lic H

ealt

h A

dvis

ory

Com

mit

tee

2006

).

• ‘P

ublic

hea

lth

is th

e or

gani

sed

resp

onse

by

soci

ety

to p

rote

ct

and

prom

ote

heal

th, a

nd to

pre

vent

illn

ess,

inju

ry a

nd

disa

bilit

y. T

he s

tart

ing

poin

t for

iden

tify

ing

publ

ic h

ealt

h is

sues

, pro

blem

s an

d pr

iori

ties

, and

for d

esig

ning

and

im

plem

enti

ng in

terv

enti

ons,

is th

e po

pula

tion

as

a w

hole

, or

popu

lati

on s

ub-g

roup

s.’ (

Nat

iona

l Pub

lic H

ealt

h Pa

rtne

rshi

p 20

06).

(See

als

o Ev

�den

ce-b

ased

pub

l�c h

ealth

.)


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

Rel�a

b�l�t

yRe

liabi

lity

is th

e ex

tent

to w

hich

a m

easu

re, w

hen

used

repe

ated

ly in

th

e sa

me

way

, will

pro

duce

the

sam

e or

a s

imila

r res

ult.

Fo

r exa

mpl

e, if

we

wer

e in

tere

sted

in m

onito

ring

a p

erso

n’s

wei

ght a

nd u

sed

a ch

eap

wei

ghin

g m

achi

ne th

at p

rodu

ced

a di

ffer

ent r

eadi

ng e

ach

tim

e it

was

use

d (w

hen

all o

ther

fact

ors

wer

e th

e sa

me)

, the

n th

e m

achi

ne w

ould

not

be

a re

liabl

e in

stru

men

t to

use.

Robu

stne

ss to

w

�thst

and

scru

t�ny

It is

impo

rtan

t tha

t ass

essm

ents

abo

ut th

e ef

fect

iven

ess

of p

ublic

he

alth

pro

gram

mes

fund

ed b

y go

vern

men

t mon

ey b

e ab

le to

w

iths

tand

scr

utin

y by

all

the

stak

ehol

ders

and

oth

er in

tere

sted

pa

rtie

s.

We

reco

mm

end

that

for m

ost p

ublic

hea

lth

prog

ram

mes

it is

ad

visa

ble

that

a p

rogr

amm

e’s

succ

ess

not b

e at

trib

uted

to a

si

ngle

cri

teri

on: r

obus

tnes

s is

pro

vide

d by

the

tota

lity

of th

e ou

tcom

e m

onito

ring

regi

me.

Sens

�t�v�

tyS

ensi

tivi

ty re

fers

to h

ow w

ell a

mea

sure

is a

ble

to a

ccur

atel

y de

tect

w

hen

a ch

ange

has

occ

urre

d in

the

outc

ome

bein

g m

onito

red.

For e

xam

ple,

a w

eigh

ing

mac

hine

that

was

abl

e to

diff

eren

tiate

a

chan

ge in

wei

ght b

y 10

0 gr

am g

radu

atio

ns is

mor

e se

nsit

ive

than

a

mac

hine

that

cou

ld o

nly

dete

ct c

hang

es in

500

gra

m g

rada

tion

s.

Targ

etA

targ

et is

the

amou

nt o

f cha

nge

desi

red,

or e

xpec

ted,

in a

spe

cific

ti

me

peri

od th

roug

h th

e im

plem

enta

tion

of t

he h

ealt

h pr

ogra

mm

e.

Targ

ets

shou

ld b

e ba

sed

on s

peci

fic a

nd m

easu

rabl

e ch

ange

s in

hea

lth

outc

omes

. Set

ting

targ

ets

help

s de

fine

the

proc

ess

tow

ards

ach

ievi

ng th

e he

alth

obj

ecti

ve s

et, a

ltho

ugh

targ

ets

requ

ire

the

exis

tenc

e of

an

appr

opri

ate

indi

cato

r (N

utbe

am 1

998)

.

(See

Ind�

cato

rs.)

Theo

ry o

f cha

nge/

act�o

nA

theo

ry o

f cha

nge/

acti

on is

a s

tate

men

t abo

ut h

ow p

ract

itio

ners

be

lieve

indi

vidu

al, i

nter

-gro

up a

nd s

ocia

l/ s

yste

mic

cha

nge

happ

ens,

and

how

the

prop

osed

act

ions

will

pro

duce

pos

itiv

e re

sult

s. A

goo

d th

eory

of c

hang

e ex

plic

itly

des

crib

es th

e as

sum

ptio

ns a

nd c

ausa

l lin

ks th

ough

t to

be im

port

ant f

or a

chie

ving

th

e he

alth

out

com

e so

ught

.

Whe

re in

itia

l evi

denc

e fo

r the

eff

ecti

vene

ss o

f an

inte

rven

tion

is

lack

ing,

a ro

bust

theo

ry o

f cha

nge

shou

ld b

e ar

ticu

late

d to

ju

stif

y th

e ch

oice

of i

nter

vent

ion.

Use

d in

this

way

, a th

eory

of

chan

ge p

rovi

des

the

‘logi

c m

odel

’ fra

mew

ork

to p

lan

and

just

ify

the

inte

rven

tion

sel

ecte

d. A

theo

ry o

f cha

nge

appr

oach

may

be

part

icul

arly

app

ropr

iate

whe

n in

terv

enti

ons

are

aim

ed a

t min

orit

y po

pula

tion

gro

ups.


Term

Defi

n�t�o

nEx

plan

ator

y co

mm

ent a

nd e

xam

ple

T�m

ely

data

/�n

form

at�o

nIn

form

atio

n is

tim

ely

whe

n it

cont

ains

dat

a di

rect

ly re

leva

nt to

the

prog

ram

me

and

– im

port

antl

y –

ava

ilabl

e in

a ti

me

fram

e w

here

it

can

usef

ully

info

rm d

ecis

ions

abo

ut w

heth

er th

e pr

ogra

mm

e is

pe

rfor

min

g as

pla

nned

and

wha

t thi

ngs

need

to b

e ch

ange

d to

im

prov

e th

e pe

rfor

man

ce o

f the

pro

gram

me,

if re

quir

ed.

Som

e na

tion

al d

ata

sets

are

sch

edul

ed to

be

colle

cted

on

5-ye

arly

dat

a. S

uch

data

is u

nlik

ely

to b

e ti

mel

y fo

r per

form

ance

m

onito

ring

pur

pose

s if

the

prog

ram

me

will

onl

y ru

n fo

r 3 y

ears

, sa

y. S

imila

rly,

nat

iona

l mor

talit

y an

d m

orbi

dity

dat

a is

typi

cally

on

ly a

vaila

ble

18 to

24

mon

ths

afte

r the

eve

nt. S

uch

info

rmat

ion

may

not

be

tim

ely

for s

ome

prog

ram

mes

, alt

houg

h lo

cal d

ata

may

be

ava

ilabl

e w

ithi

n a

mor

e us

eful

tim

e fr

ame.

T�m

�ng

Tim

ing

refe

rs to

the

tim

e pe

riod

bet

wee

n w

hen

an a

ctiv

ity

took

pla

ce

and

whe

n a

chan

ge in

the

desi

red

outc

ome

was

obs

erve

d.

Tim

ing

is a

n im

port

ant a

spec

t in

esta

blis

hing

‘att

ribu

tion

’, an

d be

ing

able

to m

onito

r whe

ther

a p

rogr

amm

e is

wor

king

.

Val�d

�tyVa

lidit

y re

fers

to h

ow w

ell a

pro

pose

d in

stru

men

t mea

sure

s th

e ou

tcom

e of

inte

rest

. Som

e ou

tcom

es m

ay b

e m

easu

red

dire

ctly

, w

hile

oth

ers

will

requ

ire

a pr

oxy

met

hod

that

resu

lts

in a

pro

xy

mea

sure

. Gen

eral

ly, t

he m

ore

dire

ct a

mea

sure

is, t

he m

ore

valid

it is

lik

ely

to b

e.

Ther

e is

oft

en m

ore

than

one

way

of m

easu

ring

an

outc

ome

of

inte

rest

, and

ther

e ca

n be

con

side

rabl

e de

bate

abo

ut h

ow ‘v

alid

’ a

mea

sure

act

ually

is.

For e

xam

ple,

the

best

way

to m

easu

re b

ody

com

posi

tion

(par

ticu

larl

y le

an a

nd fa

t mas

s) is

und

er-w

ater

den

sito

met

ry

(wei

ghin

g), b

ut th

is is

not

pra

ctic

al o

utsi

de s

mal

l clin

ical

stu

dies

.

Ther

efor

e, m

easu

res

of w

eigh

t adj

uste

d fo

r hei

ght (

eg, b

ody

mas

s in

dex,

BM

I) a

re o

ften

use

d to

refle

ct b

ody

com

posi

tion

,

part

icul

arly

bod

y fa

t mas

s. A

ltho

ugh

BM

I is

corr

elat

ed w

ith

body

fat m

ass,

this

rela

tion

ship

var

ies

acco

rdin

g to

bod

y bu

ild, a

ge

and

ethn

icit

y. T

here

fore

, BM

I is

not r

ecom

men

ded

for a

sses

sing

exce

ss b

ody

fat m

ass

(obe

sity

) at a

n in

divi

dual

leve

l.

How

ever

, BM

I can

be

usef

ul fo

r ass

essi

ng o

besi

ty a

t a p

opul

atio

n

leve

l if u

sed

and

inte

rpre

ted

appr

opri

atel

y. B

MI b

ased

on

dire

ct

mea

sure

men

t of h

eigh

t and

wei

ght i

s m

ore

valid

than

BM

I bas

ed

on s

elf-

repo

rted

hei

ght a

nd w

eigh

t, p

arti

cula

rly

if m

easu

rem

ents

are

take

n by

trai

ned

obse

rver

s us

ing

appr

opri

ate

equi

pmen

t and

stan

dard

ised

pro

cedu

res.

Ult

imat

ely,

the

choi

ce o

f mea

sure

for a

par

ticu

lar p

rogr

amm

e w

ill

depe

nd o

n a

num

ber o

f fac

tors

, inc

ludi

ng v

alid

ity

and

prac

tica

lity.


References Bartholomew L, Parcel G, Kok G, et al. 2006. Planning Health Promotion Programmes: An intervention mapping approach. San Francisco. Jossey-Bass.

Beaglehole R, Bonita R, Horton R, et al. 2004. Public health in the new era: improving health through collective action. Lancet 363: 2084–6.

Brownson RC. 2003. Evidence-based Public Health. Oxford: Oxford University Press.

Davidson J. 2005. Evaluation Methodology Basics: Nuts and bolts of sound evaluation. Thousand Oaks. Sage.

Department of Health. 2004. Public Health Electronic Library Standards for Better Health. London: Department of Health. London.

Iverson A. 2003. Preparing Program Objectives: Theory and practice. Toronto: The International Development Research Centre Evaluation Unit.

Jenicek M. 1997. Epidemiology, evidenced-based medicine, and evidence-based public health. Journal of Epidemiology. Dec; 7(4): 187–97.

Kohatsu ND, Robinson JG, Torner JC. 2004. Evidence-based public health: an evolving concept. American Journal of Preventive Medicine 27: 417–21.

Ministry of Health. 2005. The New Zealand Health Monitor: Updated strategic plan. Wellington: Ministry of Health.

Ministry of Health. 2006. A Guide to Developing Public Health Programmes: A generic programme logic model. Wellington: Ministry of Health.

Ministry of Health. 2007. Leading for Outcomes. http://www.leadingforoutcomes.org.nz Wellington: Ministry of Health.

National Public Health Partnership. 2006. Public Health Classifications Project Phase One: Final report. Melbourne: National Public Health Partnership.

Nutbeam D. 1998. Health promotion glossary. Health Promotion International 13: 349–64.

Public Health Advisory Committee. 2006. Health is Everyone’s Business: Working together for health and wellbeing: A report to the Minister of Health on the implications of a changing context for public health in New Zealand. Wellington: Public Health Advisory Committee.

The Treasury, State Services Commission. 2007a. Guidance and Requirements for Crown Entities: Preparing the Statement of Intent. Wellington: The Treasury, State Services Commission.

The Treasury, State Services Commission. 2007b. Guidance and Requirements for Departments: Preparing the Statement of Intent. Wellington: The Treasury, State Services Commission.


The Treasury, State Services Commission, Department of the Prime Minister and Cabinet. 2005. Guidance for Crown Entities: Planning and managing for results. Wellington: The Treasury, State Services Commission, Department of the Prime Minister and Cabinet.

United Nations Development Programme Evaluation Office. 2002. Handbook on Monitoring and Evaluating for Results. New York: United Nations Development Programme Evaluation Office.

World Health Organization. 1998. Health Promotion Glossary. Geneva: World Health Organization.

Further read�ngErnst K. 2004. The Canadian Outcomes Research Institute’s Program Logic Model. Calgary: The Canadian Outcomes Research Institute.

European Research Group Health Outcomes. 1996. Choosing a health outcomes measurement instrument: general advice proposed by the European Research Group on Health Outcomes Measures (ERGHO). Quality of Life Newsletter 15: 7–8. URL: http://www.mapi-research.fr/pdf/newsletter/qol15 0.PDF. Accessed 3 November 2005.


Documents

How to Mon tor for Populat on Health Outcomes: Gu del nes ...€¦ · by analysing population health outcomes and risks and determinants, and examining inequalities across regional