Consumer-Operated Service Programs Results

Jean Campbell, Ph.D.Greg Teague, Ph.D.

E. Sally Rogers, Sc.D.Asya Lyass, M.S. Ph.D. (Cand.)

FROM INNOVATIONS TO PRACTICE: THE PROMISE AND CHALLENGE OF

ACHIEVING RECOVERY FOR ALLHyatt Regency Hotel in Cambridge, MA., on April 14, 2008

2

Background

Over the past three decades peer-run support services have matured, diversified, increased in numbers across the United

States

3

What is a COSP?

A Consumer–Operated Services Program is peer-run service program that is administratively controlled and operated by mental health consumers and emphasizes self-help as its operational approach.

Today, most COSPs operate as an adjunct to traditional mental health services within the continuum of community care.

4

COSP Service Models

Mutual Support Groups Drop-in Centers Education and Advocacy Programs Multi-Service Agencies Specialized Supportive Programs Warm Lines

5

Number of COSP Services

SAMHSA estimates there are 7,467 groups, organizations, and services run by and for mental health consumers and/or families in the United States (2002) 44.4% are mental health mutual support groups 40.4% are mental health self-help organizations 15.2% are consumer-operated service programs

6

COSP Effectiveness

Studies using non-randomized control groups or pretest scores as comparisons found that participation in peer support Reduces psychiatric symptoms Decreases hospitalization Enlarges social networks Enhances self-esteem and social functioning

7

COSP Effectiveness

Recent studies using randomized control groups and pretest scores as comparisons have found that participation in peer support promotes wellness (a sense of well-being). Hope Empowerment Social connectedness Meaning in life Self-esteem/personhood

8

COSP Study Overview

Eight program sites CA, CT, FL, IL, ME, MO, PA, TN

Three general program models Drop-In (4 sites) Mutual Support (2 sites) Education/Advocacy (2 sites)

9

Participating Study Sites

10

COSP Study Overview

One-year longitudinal follow-up 4 measurement points: 0, 4, 8, 12 months

Participants Persons 18+ with diagnosable mental / behavioral /

emotional disorder and functional impairment N = 1827 enrolled in study; 1600+ in analysis

Common interview protocol Logic Model Conventional RCT approach

Intent-to-treat analysis Optimized, common a priori hypothesis

11

Primary Hypothesis

Participants offered both traditional and consumer-operated services would show greater improvement in well-being over time than participants offered only traditional mental health services.

12

Well-Being Outcome Rationale

To develop a measure that was supported by theory and peer literature

Hypothesized to be most sensitive to primary peer support program effect

Short term outcome Realization of participants that “We are not alone.”

13

Composite Well-being Measure: Constituent Scales

Scales incorporated in Well-being measure (WB2) Total Herth Hope Index Meaning of Life Framework Subscale Empowerment / Making Decisions (EMD) Self-

esteem/ Self-efficacy Subscale Recovery Assessment Scale (RAS) Goal Subscale Recovery Assessment Scale (RAS) Hope Subscale

Internal consistency Cronbach’s alpha = .92 at all four measurement

points

14

Results: Well-being Over Time by Random Assignment Group

WB2 by Assignment Group - Overall

-.2

-.1

.0

.1

.2

.3

Base(N=1600)

4mo(N=1441)

8mo(N=1357)

12mo(N=1272)

TMHS, 797

COS+, 803

15

Intent-To-Treat Results for Well-being – All Sites

Increase in well-being over time for participants overall

Possible differences in change in well-being over time across sites

Significantly greater increase in well-being for persons offered use of COSP (p < .05) overall

Possibly important differences in this effect across site

16

Well-being Effect Sizes By SiteWB2 Effect Sizes by Site

-.4

-.2

.0

.2

.4

.6

1 2 3 4 5 6 7 8

ES

17

Well-being Effect Sizes By Site: Between Conditions & Within COSP+

WB2 Effect Sizes: Time, Condition by Site

-.4

-.2

.0

.2

.4

.6

1 2 3 4 5 6 7 8

ES-C

ES

18

Well-being Effect Sizes By Site: Between and Within Conditions

WB2 Effect Sizes: Time, Condition by Site

-.4

-.2

.0

.2

.4

.6

1 2 3 4 5 6 7 8

ES-T

ES-C

ES

19

Intent-To-Treat Results for Well-being – Seven Sites

Increase in Well-being over time for participants overall

No significant site difference in change in Well-being over time – very consistent positive ES for COSP across all 7 sites, along with positive trend for both groups

Significantly greater increase in Well-being for persons offered use of COSP (p < .01)

Negligible differences in this effect across site Site variations in experimental effect for this

outcome are driven by variations in TMHS programs

20

As-Treated Analysis Engagement rates and adherence to assigned condition

low 57% of participants assigned to COS+ used COS 15% of participants assigned to TMHS only used COS

Consequently, ITT results may be too conservative

Scores calculated for propensity to use COS

Analysis limited to middle third of propensity Removed those who especially seek/avoid COS Effects can be attributed to intervention, not selection Smaller N means positive findings are conservative

21

As-Treated Analysis: Two Measures of Engagement in COSP

Binary measure of simple engagement – visited at least once vs. didn’t use at all

Three-point measure of intensity of use No use Low use = less than study median (8.5 visits to

COSP over 12-month period) High use = more than study median

22

Well-being Over Time By Engagement (Middle 1/3 Propensity)

WB2 over Time by Engagement Group

-.2

-.1

.0

.1

.2

.3

Base(N=516)

4mo(N=479)

8mo(N=449)

12mo(N=404)

TMHS, 331

COS+, 185

23

Well-being By Intensity of COSP Use (Middle 1/3 Propensity)

WB2 by COS Use (Middle Propensity Group)

-.2

-.1

.0

.1

.2

.3

Base(N=516)

4mo(N=479)

8mo(N=449)

12mo(N=404)

High, 77

Low, 79

None, 360

24

COSP-Findings for Well-being

Significant gains in well-being were found for the group that was offered use of COSP (the formal experiment – intent-to-treat analysis)

Greater gains in well-being were found for the group of participants who actually used COSP services (as-treated analysis)

Greatest gains in well-being were found for the group of participants who used COSP the most, those in the upper half of frequency of use

Variations in effects across sites were unrelated to formal COSP type

25

Sub-Study: How is empowerment effected by involvement in consumer run programs

Why study empowerment?Many programs espouse empowerment as both

an aspect of the program structure (“empowering practices”) and as an outcome to be achieved

However, few rigorous studies of the effects of consumer run programs on empowerment exist

Most studies not randomized, small, descriptiveComplicating matters: there is no consensus of

the definition of empowerment

26

Sub-Study: Empowerment

Making Decisions Empowerment (MDE), 28 item scale to measure subjective feelings of empowerment: self-efficacy, perceived power, optimism about and control over the future, and community activism

Personal Empowerment (PE), 20 item tool with 2 subscales: Choice and Reduction in Chance

Organizational Empowerment (OME), 17 item scale about involvement in an community, organization or club

27

Purpose

Determine whether COSP sites varied with respect to differences between E & C or change over time

Describe pattern in change over time

Examine whether “engagement intensity” (binary or no, low, high use) resulted in greater changes in empowerment

28

Intent to Treat Results Results using strict (ITT) approach yielded marginal

results--perhaps due to modest engagement in the COSP

Results of ITT analyses with both measures of personal empowerment had overall results below threshold for significance and very small effect sizes

As with well-being analyses, one site obscured the more positive results from remaining sites.

Without this site, both measures showed a significant, small and positive effects

There was still significant cross-site variation remaining for PE Choice

29

Analysis of differences in slopes over time

No difference between groups over time

0

2

4

6

8

10

0 1 2 3 4 5 6

timeou

tcom

e C

E

Difference between groups over time

0

5

10

15

20

25

30

0 1 2 3 4 5 6

time

outc

ome C

E

30

Effect sizes for the difference in slopes between the experimental and the control group over time - MD

Site 1

Site 2 Site 3 Site 4 Site 5 Site 6 Site 7

Site 8

All

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

Eff

ec

t S

ize

N=109

N=561

N=118

N=224

N=222N=143

N=133

N=93

N=1603

31

Differences between groups over time-personal empowerment

Sample items: “I can pretty much determine what will happen in my life” “People working together can have an effect on their community”

Half of the sites showed a positive difference between E and C groups in personal empowerment (MD) over time with an effect size of .2 or more

One site had a significant negative difference in change over time between E and C

One site had slight negative difference; remaining had no difference

Conclusion: positive effect on personal empowerment from participation in COSP

32

Effect sizes for the difference in slopes between the experimental and the control group over time - PE (Choice)

Site 1 Site 2 Site 3

Site 4

Site 5

Site 6

Site 7

Site 8

All

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Eff

ec

t S

ize

N=113

N=115

N=559 N=225

N=223

N=143

N=133

N=93

N=1604

33

Differences between groups over time- PE-Choice

Sample item: “How much choice do you have about how you spend your free time”

Three sites had positive difference in change over time between E and C groups with effect sizes above .2

Five sites had a negative difference in change over time, or no change

Conclusion: Some COSP affect perception of choice

34

Effect sizes for the difference in slopes between the experimental and the control group over time - PE (Reduction in chance)

Site 1

Site 2

Site 3

Site 4

Site 5

Site 6

Site 7

Site 8

All

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Eff

ec

t S

ize

N=113

N=555 N=117 N=222 N=219 N=143

N=135

N=90 N=1594

35

Differences between groups over time-PE-Reduction in Chance

Sample item: “How likely is it that you will get enough to eat in the next month”

Two sites had positive differences in change over time between E and C

Six sites had no positive differences in change over time between E and C, or negative differences

Conclusion: Perception of life being left to chance or control over life not affected by COSP

36

Organizational Empowerment There was a general downward trend in

organizational empowerment but significant variation across sites

There was less decline among COSP programs than traditional programs

Data raises questions about the fit of construct “organizational empowerment” with COSP programs

Conclusion: Relatively few participants (in either condition) experienced an increase in the number of organizational roles or activities tapped in this measure.

37

As treated analyses

Needed to go beyond ITT analyses because of modest engagement in the COSP for experimental folks

Using a procedure described earlier and propensity scores, created two groups of balanced individuals

Individuals in both groups showed equal tendency to attend COSP

Yielded a better comparison of E and C participants

38

Results

Results of the as-treated analyses confirmed the inferences that emerged from the ITT analyses:

Use of consumer-operated services was positively associated with increases in personal empowerment as measured by both the MD and the PE Choice scales, results that held without significant variation across the eight sites. Gains from using COSPs become apparent only with higher levels of use.

39

Conclusions

Results support the conclusion that COSPs in general have a positive impact on aspects of empowerment (MD and PE Choice scales)

Some COSPs have the effect of improving empowerment, while others are less effective

Effects significant but small in magnitude and consistent with other studies

40

Conclusions

Individuals with greater engagement in and attendance at COSPs fared better in their personal empowerment outcomes

Additional analyses suggested positive changes in self-efficacy items: “I believe I am a person of worth”; “I am usually confident about the decisions I make”.

COSP may have more of an effect on self-efficacy than other aspects of personal empowerment

41

What is Fidelity?

Fidelity measures provide an objective rating system to assess the extent that components of a program are faithfully implemented according to intended program model, theory, or philosophy.

42

Why Measure Fidelity?

The use of fidelity measures has become a widely accepted methodological tool in mental health services research and serves a number

of important purposes.

43

Why Measure Fidelity?

In addition to establishing that a set of well-defined services leads to predicted outcomes, it is critical for researchers to establish the

integrity of the service delivery. Providers need to adhere to critical elements

of an evidence-based practice in order to achieve the positive outcomes identified in the original research.

44

COSP Fidelity Measurement

Common measurement, diverse programs Analytic challenges

Some theoretically important program aspects were not being measured

Comparison conditions were highly variable Initial program measurement goal

Develop cross-site program-level implementation measure

Assess interventions within traditional pooled data framework

45

Fidelity Assessment Common Ingredients Tool: FACIT

Process of Developing FACIT Identification of common ingredients Definition of common ingredients Feasible performance indicators (48 items) Performance anchors (typically 4-5) Involvement of COSP directors and staff as well as

researchers/evaluators at all stages of FACIT development process

Involvement of the CAP in the definition of common ingredients

46

FACIT Operationalization

Data collection During site visits, questions devised to elicit

information about common ingredients from program directors and program staff.

COSP recipient focus group Program observation

While FACIT was developed to measure characteristics thought to be common to COSPs, it was also used to measure the extent to which TMHS were “consumer-friendly.”

47

FACIT Operationalization

Independent rating After site visits, site visitors independently rated each

program on each dimension.

Conciliation Site visitors came to agreement on any dimensions on

which there was disagreement.

48

FACIT Operationalization

Pilot testing of FACIT (Round 1 site visits) Average interrater reliability of items .8; use of FACIT

feasible with both COSP and Traditional Mental Health Services

FACIT Psychometrics CIs present within COSPs Differences between COSPs and TMHS detected Differences among COSP models

49

FACIT Operationalization

COSP Fidelity Measurement (Round 2 site visits)

Analysis & further psychometrics Factor analysis and internal consistency within

major scales Identification of subscales for use in fidelity-

outcome analyses

50

FACIT Scales and SubscalesUsed in COSP Study

STRUCTURE Consumer Ownership Responsiveness

ENVIRONMENT Inclusion Accessibility

BELIEF SYSTEMS Peer Ideology Choice & Respect Spirituality & Accountability

PEER SUPPORT Encouragement Self-Expression

EDUCATION ADVOCACY

51

FACIT Scale Scores: COSP and Traditional Services

0 20 40 60 80 100

ADVOCACY

EDUCATION

PEER SUPPORT

BELIEFSYSTEMS

ENVIRONMENT

STRUCTURE

% of possible subscale total

COSP

TMHS

52

All overall FACIT scores for COSP were significantly higher than the score for any TMHS

Mean COSP scores were higher than TMHS on all main subscales

COSP scored higher than their respective TMHS on most subscales

FACIT Scores: COSP vs. Traditional Services

53

Comparing Fidelity & Outcome (COS+ vs. TMHS – 8 sites)

Sample Standardized Outcome and Fidelity By Site

-2

-1

0

1

2

1 2 3 4 5 6 7 8

s.d

.

Empowerment (MD) Mean of FACIT Process Scales

54

Mean Correlations: FACIT Scales and Selected Outcomes (8 Sites)

Mean Correlations of FACIT Scales With Outcomes(WB2, MD, Social Inclus & QOL, Life Sat)

-.4 -.2 .0 .2 .4 .6 .8

ADVOCACY

EDUCATION

PEER SUPPORT

BELIEF SYSTEMS

ENVIRONMENT

STRUCTURE

Pearson r

55

FACIT Process Scales & Selected Outcomes: Between & Within Programs

Correlation Coefficients: Mean FACIT Process Scores & Selected Outcomes - Experimental & Observational

.0

.1

.2

.3

.4

.5

.6

.7

.8

Emp (MD) Wellbeing Socializ Life Sat

Pe

ars

on

r

COS+ vs T-only All 16 programs over time

56

FACIT Process Scales and Specific Outcomes

FACIT Process Scales & Selected Outcomes (N=8)

.0

.2

.4

.6

.8

1.0

ENVIRONMENT BELIEF SYSTEMS PEER SUPPORT

Pe

ars

on

r

Empowerment Wellbeing Socializ Life Satisf

57

Key Ingredients Related To COSP Outcomes: Scales & Subscales

Environment Inclusion subscale: Low/no cost; protective

program rules; positive social environment; sense of community; lack of coerciveness

Belief Systems Choice & Respect subscale: Choices about

participation; acceptance and respect for diversity Peer Support

Self-Expression subscale: Artistic expression; participants telling own stories

58

Summary: Fidelity and Outcome Analysis COSP are effective in producing gains in

recovery-related domains over a 12-month period

Gains in these areas are strongly related to specific program (fidelity) ingredients, measured at the program level

Critical ingredients are stronger in COSP but are present in traditional programs as well

Critical ingredients related to other outcome domains, independent of program type (COSP TMHS) or experimental effect

59

COSP-Substantive Findings Evidence base for COSP as discrete

programs Adding COSP to traditional services adds to

well-being Effect is both incremental and compensatory:

impact is strongest where TMHS programs are weakest

Evidence base for recovery theory Program features specified for and found in

COSP are related to increases in well-being independent of setting

60

COSP-Policy Implications

Results provide evidence of COSP effectiveness

Results reinforce commitment to recovery–oriented services Particular consumer-supported program features

contribute to recovery and are effective across diverse settings

Program features central to COSP should also be fostered within TMHS

61

COSP Next Steps

COSP Recognized by SAMHSA as an EBP COSP Evidence-Based Practices Kit

developed KIT materials organized around FACIT domains and

common ingredients FACIT protocol available

KIT in field review

62

COSP Next Steps

“Raising All Boats” Initiative part of Missouri DMH Transformation Efforts FACIT implemented at 5 Drop-in Centers E-FACIT and Users’ Guide developed COSP ToolKIT tested in program training FACIT revised for piloting at 5 Warmlines FACIT results used to guide CQI efforts

6363

Using the e-FACIT As a Self-Assessment CQI Tool

Easy Administration (2 persons) Interviews with scheduled questions from protocol Focus groups Observation

Easy Scoring 48 items Anchored scoring

Easy Data Entry Excel 2003 program

Easy Charting The E-FACIT Workbook developed to automatically

produce charts

6464

Using the e-FACIT As a Self-Assessment CQI Tool

Conduct annual assessments CQI Team Outside evaluator

Share results with staff & membership Target areas to improve Develop Quality Improvement Plan

Implement COSP Toolkit (targeted areas) Revise or enhance programmatic approaches Augment staff supervision & training