Presented by Gregory B. Teague, Ph.D. Matthew Johnsen, Ph.D. Joseph Rogers Bonnie Schell, M.A. (See additional credits at end) Research on Consumer-Operated

  • View

  • Download

Embed Size (px)

Text of Presented by Gregory B. Teague, Ph.D. Matthew Johnsen, Ph.D. Joseph Rogers Bonnie Schell, M.A. (See...

  • Slide 1

Presented by Gregory B. Teague, Ph.D. Matthew Johnsen, Ph.D. Joseph Rogers Bonnie Schell, M.A. (See additional credits at end) Research on Consumer-Operated Service Programs: Effectiveness Findings and Policy Implications of a Large Multi-Site Study Findings from the SAMHSA/CMHS Consumer-Operated Service Program Multisite Research Initiative Slide 2 2 - 1 - Consumer-Operated Services Multisite Research Initiative: Overview Slide 3 3 Consumer-Operated Services: An Invocation "Who then can so softly bind up the wound of another as he who has felt the same wound himself? Thomas Jefferson Slide 4 4 Consumer-Operated Services: Context Peer-delivered services for persons with serious mental illness have grown in number and diversity in recent years There is growing recognition of their value (cf. IOM Quality Chasm and New Freedom Commission reports) There is preliminary but limited evidence of their effectiveness in improving symptoms, promoting larger social networks, and enhancing quality of life Slide 5 5 Context (contd.) However, evidence has been limited to uncontrolled studies, demonstrations of feasibility, & preliminary findings. Weak evidence-base has hindered peer-run program credibility, resource stability & sustainability, & opportunities for integration into the continuum of community care. Slide 6 6 COSP Multisite Research Initiative: Design Overview Research question: To what extent are consumer-operated programs effective as an adjunct to traditional mental health services in improving the outcomes of adults with serious mental illness? Experimental design: Random assignment to one of two conditions Traditional MH Services (TMHS) or TMHS+COSP Parallel cost study Consumer involvement at all levels Slide 7 7 COS Criteria for COSP-MRI Administratively controlled and operated by consumersplanned, delivered, and evaluated by consumers themselves In full operation for at least 2 years Does not include all types of peer-run and self-help services Slide 8 8 Design Overview (contd) Participants Persons 18+ with diagnosable mental / behavioral / emotional disorder and functional impairment N = 1827 enrolled in study Eight program sites CA, CT, FL, IL, ME, MO, PA, TN Three general program models Drop-In (4 sites) Peer Support (2 sites) Education/Advocacy (2 sites) Began in 1998 Slide 9 9 Design Overview (contd) One-year longitudinal follow-up 4 measurement points: 0, 4, 8, 12 months Common interview protocol Outcome domains Employment, Empowerment, Housing, Service Satisfaction, Social Inclusion, Symptoms, Well- being Conventional RCT approach Intent-to-treat analysis Optimized, common a priori hypothesis Slide 10 10 Primary Hypothesis Informed by presumed underlying consumer experience: I am not alone Theoretically justified by literature on well- being and research on peer support programs and consumer/survivor recovery The well-being construct was developed from the validated scales in the protocol which measured existential dimensions of participant experience: Recovery, empowerment, quality of life, social inclusion & acceptance, meaning of life, hope Slide 11 11 Primary Hypothesis Participants offered both traditional and consumer-operated services would show greater improvement in well-being over time than participants offered only traditional mental health services. Slide 12 12 - 2 - Program/Fidelity Measure: The Consumer Operated Services Program Fidelity Assessment Common Ingredients Tool (COSP-FACIT) Slide 13 13 Initial COSP Measurement Context Common measurement at participant level Potentially important aspects of intervention not measured in common protocol COS not operationalized; no existing measures Diversity among programs Contrast between experimental and control conditions not specified Slide 14 14 Development of the FACIT Involvement of COS directors and staff as well as researchers at all stages Identification and definition of common ingredients of consumer-operated services Involvement of the Consumer Advisory Panel Identification/selection of feasible indicators Specification of performance anchors (typically 4-5) for each indicator Slide 15 15 Hypothesized Common Ingredients of Consumer-Operated Services Structure Consumer operated Participant responsive Links to other supports Environment Accessibility Safety Informal setting Reasonable accommodation Belief Systems Peer principle Helpers principle Empowerment Choice Recovery Acceptance and Respect for diversity Spiritual growth Slide 16 16 Hypothesized Common Ingredients of Consumer-Operated Services Peer Support Peer support Telling our stories Consciousness-raising Crisis prevention Peer mentoring and teaching Education Self-management / problem-solving Education Advocacy Self-advocacy Peer advocacy Systems advocacy Slide 17 17 Sample Element: 1.1 Structure - Consumer Operated 1.1.1 Board Participation - Consumers constitute the majority on the board or the group which decides all policies and procedures. 1.1.2 Consumer Staff - With limited exception, staff consists of consumers who are hired by and operate the program. 1.1.3 Hiring Decisions - Consumers have control over hiring decisions. 1.1.4 Budget Control - Consumers have control of the operating budget. 1.1.5 Volunteer Opportunities - Role opportunities for participants may include board and leadership positions, volunteer jobs, and paid staff positions. Slide 18 18 Sample Anchors: 1.1.1 Board Participation 1No member of the board is self-identified as a consumer 2 1-50% of the board are self-identified as consumers 3 51% or more of the board are self-identified as consumers but less than 51% of the officers are self-identified as consumers 4 51% or more of the board are self-identified as consumers and more than 51% of the officers are self identified as consumers 5 90-100% of the board are self-identified as consumers and all of the officers are self-identified as consumers Slide 19 19 Application of the FACIT Data collection Two rounds of site visits years 2 and 4 Interviews with program directors, staff, & recipients in both COS and TMHS Independent ratings by site visitors Conciliation Raters established agreement following any initial disagreement Pilot testing (Round 1) Evaluated feasibility and inter-rater reliability, modified for round 2 Slide 20 20 FACIT: Psychometric Analysis Factor and internal consistency analyses within major domains Conservative elimination of weak or contrary variables Maximal retention of original variables to optimize content validity Delineation of provisional scales for use in fidelity-outcome analyses Partial overlap of hypothesized and empirically- defined constructs Slide 21 21 FACIT Research Questions Is the FACIT reliable and valid as a measurement tool? Are the Common Ingredients in fact common? Does the FACIT distinguish between consumer-operated and traditional services? Does the FACIT distinguish among models of consumer-operated services? Slide 22 22 FACIT: Inter-Rater Reliability Inter-rater reliability, assessed in Round 1 Average across all items.70 Average, all items, COSP.72 Average, all items, TMHS.67 Overall score.97 Inter-rater reliability, refined measure Average of all items retained.78 Acceptable inter-rater reliability at item level Excellent inter-rater reliability for total scale Slide 23 23 FACIT Subscales: Internal Consistency (1) (N = 16) Scale/SubscaleCr. Alpha# items STRUCTURE.939 7 Consumer Ownership.9815 Responsiveness.8112 ENVIRONMENT.738 7 Inclusion.7515 Accessibility.8992 BELIEF SYSTEMS.738 8 Peer Ideology.8024 Choice & Respect.6862 Spirituality & Accountability.6832 Slide 24 24 FACIT Subscales: Internal Consistency (2) (N = 16) Scale/SubscaleCr. Alpha# items PEER SUPPORT.885 6 Encouragement.9293 Self-Expression.8153 Self-Expression2.7602 EDUCATION.888 5 ADVOCACY.893 3 TOTAL.963 36 Slide 25 25 Overall FACIT Scores: Consumer- Operated and Traditional Services Slide 26 26 Generally high performance on most dimensions COS organizations received 75% of all possible points COS organizations received 82% of possible points on four consumer-defined organizational process scales (Factor 1 of 2-factor solution: 52% out of total 80% explained) Some variability across subscales and sites Room to increase no problem with ceiling effects Presence of Common Ingredients in COSP Slide 27 27 All overall FACIT scores for COSP were higher than the score for any TMHS (p =.004) Mean overall scores COSP76%(68% - 83%) TMHS42%(25% - 54%) Mean COSP scores were higher than TMHS on all main subscales (t-test p-values:.000 -.030) Greatest difference on Structure Least differences on Belief Systems, Peer Support, and Education COS programs scored higher than their respective TMHS on most subscales Consumer-Operated vs. Traditional Services Slide 28 28 Variability in overall scores across sites (68% - 83%) Some variation in overall scores by model type Consumer Run Drop In Centers74% Peer Support Programs75% Education & Advocacy Programs82% Greater variation in subscales across model types and sites Differences and Similarities Among COS Models Slide 29 29 Variations Among Service Types Within Domains Slide 30 30 All models show high ratings on Belief Systems Education and Advocacy programs appear higher on Education and Advocacy domains Programs that more explicitly include peer support components appear higher on Peer Support More formally structured programs appear higher on Structure and Environment Differences and Similarities Among COS Models Slide 31 31 FACIT: Conclusions The FACIT measures salient features of a wide range of consumer-operated service models The models included in the COSP study were consistent with general specifications for the common ingredients and were significantly more so than control programs Generalizability is not yet known, but there is positive evid