Measuring OutcomesMeasuring Outcomesfor quality and accountabilityfor quality and accountability

The changing landscape in mental health and

rehabilitation

The changing landscape in mental health and

rehabilitation Increasing demand for accountability

i wide range of service settingsFunding agencies demand that

providers examine service effectiveness and measure consumer outcomes

Indications of quality improvement and capacity of management

Meet core values and get input from service consumers

Increasing demand for accountability i wide range of service settings

Funding agencies demand that providers examine service effectiveness and measure consumer outcomes

Indications of quality improvement and capacity of management

Meet core values and get input from service consumers

Planning a program evaluation

Planning a program evaluation

Individual accounts and anecdotesConcrete and standardized ways

to measure service effectivenessWhat is “best practice”?

Individual accounts and anecdotesConcrete and standardized ways

to measure service effectivenessWhat is “best practice”?

Planning a program evaluation

Planning a program evaluation

Program administrators are often not well informed about outcome measurement

… and how best to link outcome measures to questions that they want to address related to services

Program administrators are often not well informed about outcome measurement

… and how best to link outcome measures to questions that they want to address related to services

The best starting point is to determine:

The best starting point is to determine:

What outcomes need to be addressed?

Who is the evaluation information being developed for?

Who will receive information about the results?

Example:

Results can be disseminated to stakeholders such as funding and accreditation bodies, a board of directors, primary consumers, or family members.

What outcomes need to be addressed?

Who is the evaluation information being developed for?

Who will receive information about the results?

Example:

Results can be disseminated to stakeholders such as funding and accreditation bodies, a board of directors, primary consumers, or family members.

A program evaluation effort can be started at any time – when a service starts or added later to an existing service

A program evaluation effort can be started at any time – when a service starts or added later to an existing service

Clarifying outcomes to be measured

Clarifying outcomes to be measured

Conducting an evaluation, study, or quality assurance effort begins with understanding which outcomes need to be addressed

Conducting an evaluation, study, or quality assurance effort begins with understanding which outcomes need to be addressed

Steps towards clarification:

Steps towards clarification:

Get an understanding of your agency’s intended areas of effectivenessExample: The agency’s stated goal is “Increase in employment”

Ask whether the services your agency offers are likely to go beyond the intended outcomes to other unintended outcomesExample: A bonus goal is “Increase in self-esteem among service recipients”

Get an understanding of your agency’s intended areas of effectivenessExample: The agency’s stated goal is “Increase in employment”

Ask whether the services your agency offers are likely to go beyond the intended outcomes to other unintended outcomesExample: A bonus goal is “Increase in self-esteem among service recipients”

Examine your agency’s mission statement, strategic plan, or results of former program evaluation efforts to get closer to your agency’s intended effects and thus the outcomes you should measure

Examine your agency’s mission statement, strategic plan, or results of former program evaluation efforts to get closer to your agency’s intended effects and thus the outcomes you should measure

Determine what domains need to be measured from these efforts (housing, employment, social inclusion, satisfaction, etc.)

Develop clear definitions or criteria for these domains so that the appropriate measures or instruments can be located

Determine what domains need to be measured from these efforts (housing, employment, social inclusion, satisfaction, etc.)

Develop clear definitions or criteria for these domains so that the appropriate measures or instruments can be located

Selecting an outcome measure or instrumentSelecting an outcome measure or instrument

Providers must have confidence in the measures

Choose high quality and appropriate measures

Look for existing measures that match the outcomes needing to be addressed, rather than using home-grown surveys or instruments developed just for the purpose at hand

Providers must have confidence in the measures

Choose high quality and appropriate measures

Look for existing measures that match the outcomes needing to be addressed, rather than using home-grown surveys or instruments developed just for the purpose at hand

Ensure that the measures have been tested and are known to be reliable and valid, so …

that you can trust what they measure, and …

that they measure what they say they should

Ensure that the measures have been tested and are known to be reliable and valid, so …

that you can trust what they measure, and …

that they measure what they say they should

Two important paradigm shifts

Two important paradigm shifts

The need for reliable and valid measures of the subjective aspects of recovery

The need for measures that were developed from a consumer perspective

The need for reliable and valid measures of the subjective aspects of recovery

The need for measures that were developed from a consumer perspective

“The times they are a-changing”

“The times they are a-changing”

Providers, practitioners, managers, administrators, and policy makers can ease pressures they face in the present and in the future by …

making use of established, high quality measures and instruments, and

clarifying which effects and outcomes ought to be measured

Providers, practitioners, managers, administrators, and policy makers can ease pressures they face in the present and in the future by …

making use of established, high quality measures and instruments, and

clarifying which effects and outcomes ought to be measured

The results from evaluations

The results from evaluations

… may be used not only to satisfy funding bodies and other stakeholders

- they will also provide useful information that can guide the direction of changes needed to create more effective ways of helping the people being served

… may be used not only to satisfy funding bodies and other stakeholders

- they will also provide useful information that can guide the direction of changes needed to create more effective ways of helping the people being served

Mission

Policy statements

Activities

Procedures Record keeping

Supervision structure

Job descriptions

Evaluation mechanisms

Evaluation is an integrated component in program development

Evaluation is an integrated component in program development