Methodology for evaluating mental health case management

Evaluation and Program Planniny, Vol. 19. No. 2, pp. 121-129, 1996 Cmvrieht CC: 1996 Elsevier Science Ltd

Pergamon SO149-7189(96)00003-l

Printed’& GreatBrttain. All rights reserved 0149-71X9/96 %15.00+0.00

METHODOLOGY FOR EVALUATING MENTAL HEALTH CASE MANAGEMENT

DEBORAH M. BRYANT

Department of Pediatrics, Vanderbilt University School of Medicine

LEONARD BICKMAN

Center for Mental Health Policy, Vanderbilt Institute for Public Policy Studies

ABSTRACT

The evaluation methodology outlined in this paper produces a detailed description of the structure and process of a case management system. Using a theory-based model, a checklist has been developed to describe the attributes of a high-quality case management system, incorporating multiple valueframes. Items on the list describe proximal outcomes logically necessary for effective case management as identiJied by various stakeholder groups. Copyright 0 1996 Elsevier Science Ltd

INTRODUCTION

Mental health case management services, promoted as the “backbone of the system of care” (Stroul & Fried- man, 1986, p. 21), are required by federal legislation’ for individuals with chronic mental illness (Robinson & Toff-Bergman, 1989). According to Weil (1985a, p. 324), “the essence of case management is the effective integration of services to meet clients’ needs”. However, case management has been likened to a Rorschach test (Schwartz, Goldman, & Churgin, 1982); interpretation of the concept depends greatly upon individual perspective. Even when legislative mandates require case management, the states have considerable latitude in defining it (Burns, Gwaltney, & Bishop, 1995).

The heterogeneity of case management programs and the frequent failure of authors to describe case management make it difficult to interpret evidence of its effectiveness. Brekke and Test (1992) noted that an explicit description is needed to determine if services are delivered as planned, to facilitate replication, to study the “critical ingredients” of a model, and to compare varying program models. Clark, Landis, and Fisher (1990) called for investigations to “confirm successful implementation of a case management system before studies of impact are pursued” (p. 228).

This paper describes the methodological principles underlying a study of the implementation and quality of case management services in the Fort Bragg Child and Adolescent Mental Health Demonstration Project

The authors wish to acknowledge the following people for their assistance in the execution of this study: the staff of the Fort Bragg Evaluation Project who participated in all aspects of data collection and analysis, the staff at Major General James H. Rumbaugh Child and Adolescent Mental Health Clinic, Roy Haberkern, M.D., and the children and their parents who participated in the Fort Bragg Project and shared much about their lives.

This research was supported by the U.S. Army Health Services Command (DADAlO-89-C-0013) as a subcontract from the North Carolina Department of Human Resources/Division of Mental Health, Developmental Disabilities and Substance Abuse Services, and a grant to Leonard Bickman (ROIMH-46136-01) from the National Institute of Mental Health. It was also supported by a training grant to Bickman from the National Institute of Mental health (T32MH19544).

For reprints or more information, contact Dr Bryant at Division of Community Pediatrics, Vanderbilt University School of Medicine, Nashville. TN 37232-2570, U.S.A.

’ P.L. 99-660 and P.L. 102-231.

121

122 DEBORAH M. BRYANT and LEONARD BICKMAN

in Fayetteville, North Carolina, hereafter referred to as the Fort Bragg Demonstration. This project, described by Bickman (1996) in this issue, provided a continuum of mental health care to military dependent children and adolescents in the Fort Bragg catchment area. Tra- ditional outpatient and acute hospital services were complemented by the intermediate services of day-treatment, in-home counseling, therapeutic homes, special- ized group homes, and 24-hour crisis management services. A multidisciplinary team planned care for children requiring services more intensive than usual outpatient treatment. The case manager was responsible for organizing the team’s reviews, insuring that services were arranged, and monitoring children’s progress with respect to their treatment plans.

The case management study, carried out as part of a larger evaluation (Bickman, Bryant, & Summerfelt, 1993), was designed in accordance with the following principles:

??The case management intervention should be explicitly described.

??The evaluation should be guided by theory. ??A multimethod, multiinformant technique should be

used to obtain a rich description incorporating multiple perspectives.

*The criteria for assessing quality should be explicitly stated.

The following sections discuss the application of these principles and the instruments used in the study. The results of the evaluation of treatment planning, a part of case management, will be used to illustrate the methodology.

METHODOLOGIC PRINCIPLES

The Case Management Intervention Should he Explicitly Described Many researchers have called for adequate descriptions of case management services (Belcher, 1993; Graham & Birchmore-Timney, 1989; Rosenheck, Neale, & Gallup, 1993; Rubin, 1992). However, with rare exceptions (Berkowitz, Halfon, & Klee, 1992; Marcenko & Smith, 1992), the published studies of case management services for children have not explicitly described the intervention. Researchers in two children’s mental health projects are currently working to define case management for severely emotionally disturbed children. The staff of the Oregon Partners Project (DeChillo, personal communication) and researchers in the New York Office of Mental Health (Armstrong & Evans, 1992) are utilizing activity logs, questionnaires, management information system data, and in the latter project, provider record reviews.

Although there is much to learn about case man-

agement, there is general agreement on its primary components: identification, assessment, planning, linking and coordinating, monitoring and evaluation, and advocacy (Melville, Kiber, bz Haddle. 1977; Rothman. 199 I). For each component, specific intervention activities can be described. There are differences in the extent to which the various components are emphasized in different case management models (Burns et al.. 1995). For example. Weil (1985a) distinguished between a brokerage model in which the case manager con- centrates on coordination of services for the client and a primary therapist model in which the focus is on counseling and treatment. An in-depth discussion of the various models of case management is beyond the scope of this paper. but reviews are available (Burns et al.. 1995: Holloway, 1991; Robinson & Toff-Bergman, 1989; Weil, 1985b).

Optimally, specific intervention activities are planned deliberately and in accordance with a rational theory. In the real world. these activities are constrained by limited resources and unexpected barriers. Therefore. the evaluation of a case management program must document the extent to which planned activities are actually implemented. Case management was considered central to the continuum of care offered at the Fort Bragg Demonstration (Bickman, Heflinger, Pion. & Behar, 1992). It was expected that improved outcomes would result in part from a well implemented case management system. Thus, it was important to document at the conclusion of the evaluation whether actual practice matched the planned system.

The Evaluation Should be Guided by Theory Hypotheses about adequate implementation and expected outcomes should flow from a program theory (Bickman, 1987; Chen & Rossi, 1983). Peterson and Bickman (1992) proposed the use of specific content theories to formulate testable hypotheses about the components of an intervention. They noted that these hypotheses may be tested using either “traditional hypothesis-testing analyses or matching approaches comparing actual with ideal (model) service provision” (p. 168). The latter approach was used in this evaluation.

Thus, prior to studying case management services at the Fort Bragg Demonstration. the evaluation team needed a general theory. The important aspects of the intervention were obtained through examination of case management literature, program policies, and interviews with three separate stakeholder groups including parents, clinic administrators, and case managers. Input from these stakeholder groups was obtained through structured concept mapping (Trochim, 1989). The groups were led through sessions of brainstorming, sort- ing and rating of issues, and interpretation to produce criteria for high-quality case management (Marquart. Pollak. & Bickman, 1992). This information was syn-

Evaluating Case Management

Case Management Theory

123

Case Management Activities _ Intermediate Outcomes - Distal Outcomes

TREATMENT PLANNING: Comprehensive Initial Assessment Client & Family Participation Multi-disciplinary Team Participation

LINKAGE: Regular Communication among Client, Family,

Providers, & Case Manager Assistance with Arrangements for Treatment Single Point of Contact for Family & Providers

MONITORING: Review of Treatment Progress Review of Restrictiveness of Care

ADVOCACY: Assistance to Family in Negotiating with

Providers Respect for Child & Family Rights

More Appropriate Treatment Plan

Plan that is Acceptable to Child & Family

Timely Review of Treatment Process

Accessible Services

Efficient Use of Services

Improved Mental Health Outcomes

Quicker Recovery

Increased Client & Family Satisfaction

Reduced Costs

Figure 1. Theoretical model of the Fort Bragg Demonstration’s case management services.

thesized to construct the logic model in Figure 1, which illustrates the theoretical progression from case management components with specific activities (left) to proximal and distal outcomes

Successful proximal outcomes in the Fort Bragg Demonstration were expected to promote the distal outcomes of improved mental health with quicker recovery, increased client and family satisfaction, and reduced costs of treatment. However, successful achievement of these ultimate outcomes also depended upon other features of the Fort Bragg Demonstration, such as the availability of intermediate level services and the multidisciplinary team approach. Therefore, it was not feasible to relate the success of case management directly to the achievement of the distal outcomes. Consequently, the evaluation team concentrated on documenting the extent to which proximal outcomes were achieved.

A Multimethod, Multiinformant Technique Should be Used The multimethod, multiinformant approach, espoused by Rossi, Freeman and Wright (1979), provided detailed documentation on the content of case management intervention. In this study, questionnaires, naturalistic observation, chart reviews, and interviews were used to assure a complete description of the case management process.

There were three ways in which multiple methods and multiple informants helped to create a complete picture. For example, case managers completed logs to provide a basic description of their activities. However, some activities considered necessary for high-quality case management were carried out by other staff members, not case managers. These activities were not reflected in the case managers logs, but could be obtained through chart reviews and observation of treatment team meetings. Second, it was noted that the administrative structure and policies in the case management section could affect the practice of individual case managers. In prin- ciple, case managers should perform better in a supportive environment with clear job performance expectations, realistic workloads, and regular feedback (Intagliata, 1982). Therefore, information was collected on hiring and supervisory practices, perceptions of workload, and clerical support. Finally, because the culture of an organization may affect the conduct of its employees (Robinson & Toff-Bergman, 1989), an adaptation of Jerrell and Hargreaves’ (1991) Com- munity Program Philosophy Scale was administered to characterize the philosophical milieu of the agency. Thus, the multimethod, multiinformant approach provided more complete descriptions of the case management activities, the administrative structure of the program, and the agency culture.


The Criteria for Assessing Quality Should be Explicitly Stated While information obtained from multiple perspectives provided a detailed picture of case management, it did not assure that questions about quality would be answered. Data had to be collected about the aspects of the intervention that were important to various stakeholder groups. The evaluators had to identify, a priori.

criteria for assessing quality based on stakeholder values.

The perception of quality depends very much on the values of the observer. As Peterson and Bickman (1992. p. 166) expressed it, “quality is.. abstract. It is that which appeals to expectations and preferences-a statement of goodness or symbolic value that is not easily measured”. Further, as Kirkhart and Ruffolo pointed out (1993, p. 57) “. although values are the key ingredients that move evaluation beyond mere description, they are not explicitly articulated in many evaluation models. Making the value base of an evaluation explicit is especially important, since there are a number of possible values that might be used”.

By tapping the opinions of providers, families, and administrators through concept mapping (Trochim, 1989) an enriched view of “good” case management was obtained. This knowledge, combined with information obtained from the project contract, the clinic mission statement, and literature review, were used to formulate a checklist of quality indicators (Bickman & Peterson, 1990). The checklist described optimal case management activities and system characteristics (Figure 2). Special attention was given to the principles of the Child and Adolescent Service System Program (CASSP. Stroul & Friedman, 1986). Data were then collected from a parent survey, an expert consultant’s record review, and a network analysis for an assessment of the quality of the program.

DESCRIPTION OF INSTRUMENTS

Activity Logs As part of the evaluation, case managers kept a log of their activities for 5 days, organized by the major components of case management previously identified. Assessment, planning, linkage, and monitoring components were supplemented by counseling and advocacy. Sections were added for nonclinical support to families, crisis management, administrative activities, and personal time to assure complete data. Multiple activities were identified for each component.

Chart Review Evaluation staff reviewed chart documentation of case manager and treatment team activities for 100 children. Data on the length of case management and the number

of case managers sequentially assigned to a client were collected to assess continuity within the system. The occurrence of hospitalizations and other level-of-care changes, the frequency of team meetings, and the participation of clients. family members, and staff members in the meetings were also recorded. This information permitted assessment of the intensity of treatment team activities with respect to changes in client services.

Interviews and Document Review Semi-structured interviews were conducted by one of the authors (D.M.B.) with senior clinical staff, case managers, their supervisors, the personnel director, and the quality improvement/utilization review consultant. The interviews explored the process of assignment of clients to case managers; caseload make-up and size; hiring, credentialing, and supervision of the case managers; the role of case managers within the teams; quality improvement (QI) activities; general perceptions of the staff regarding the work environment.

The project contract. the job description for the case managers. and the case managers’ annual work plan were examined. Information about performance expectations was also obtained through review of the annual QI plan. focused Ql reviews. and the checklist used by the staff for chart reviews.

Community Program Philosophy Scale Adapted for Chil- dren’s Mental Health Services (CPPS-C) Clinical staff members completed an go-item scale with 20 subscales, adapted from Jerrell and Hargreaves’ (I 991) Community Program Philosophy Scale. Five subscales. from the Community-Oriented Programs Environment Scale (Moos. 1988), explored the organizational culture with respect to issues such as clarity of job roles and supervisory support. The remaining conceptually derived subscales examined specific practices and the manner in which mental health services were provided by the clinic staff. For example, three subscales explored the program’s emphasis on comprehensive treatment planning, family involvement in planning. and provision of care in the least restrictive environment.

Parent Survey Parents or guardians of children who received case management services were asked to respond to a survey about case management services received in the previous 2 months. The survey discussed in this issue (Brannan. Sonnichsen. & Heflinger, 1996) included questions about the parent’s perceptions about the treatment planning process; questions about the case manager’s accessibility and communication skills; questions about the parent’s general satisfaction with case management services. Parents were also asked to complete the Oregon Partners Project Service Coordination Scale

Evaluating Case Management

Checklist

?? Treatment a. Families and surrogate families are full participants in planning. b. The plan is comprehensive and individualized. c. Services are provided in the least restrictive, most normative environment that is

clinically appropriate.

0 Linkage Coorm a. Services are accessible to families. b. Staff are sensitive and responsive to cultural differences. c. The case manager mediates between the family and providers. d. The case manager keeps the family informed. e. The case manager links the family to supportive services. f. The case manager helps adolescent clients with transition to adult mental health

services.

. . ??m a. Regular follow-up is maintained. b. Team reviews are conducted in a timely manner.

?? Advocacv a. The case manager focuses on the client and family. b. The parent feels efficacious. c. The rights of the family and child are respected.

. ?? Case w Job Dw

a. Job responsibilities are clearly outlined. b. Hiring and job performance criteria are consistent with the job responsibilities. c. Actual activities are consistent with the job description. d. The authority of the case manager to arrange and contract for services is

unambiguous. e. The caseload size and the mix of client service intensity is manageable.

?? m a. Supervisory support is readily available. b. Peer support is adequate. c. Reception and clerical support are sufficient. d. Continuing education opportunities are available and appropriate.

Figure 2. Case management evaluation data checklist.

125

(DeChillo, Stuntzner-Gibson, Friesen, Paulson, & Koren, 1993). This is an 18-question instrument which explores parents’ perceptions of how well providers communicate, coordinate, and facilitate services for children.

Expert Review A child psychiatrist served as an independent consultant to review the records of a sample of children receiving case management services. The client intake evaluation and initial treatment plan were read and checked for accuracy and completeness. Next, treatment plan revisions, provider progress notes, and (where appli- cable) circumstances surrounding residential placement were studied. The expert reviewer was asked to rate his

agreement on a 5-point scale with 24 statements about the quality of treatment planning and follow-up as documented in the chart.

Network Analysis A written survey was used to assess the interorganizational relationships among mental health and other community agencies and to address the community perspective on the mental health service system as part of the Implementation Study of the Fort Bragg Evaluation Project (Heflinger, 1993). The “Assessing Local Service Systems for Children and Youth” survey (Morrissey, 1992; Morrissey & Burns, 1990) developed for the Robert Wood Johnson Mental Health Services Program for Youth was adapted for use with a military


dependent population. It was designed to assess service system effectiveness including capacity, performance, the adequacy, and quality in mental health services such as case management.

ILLUSTRATION OF THE METHODOLOGY: TREATMENT PLANNING

The evaluation of the treatment planning component illustrates the advantages and limitations of this methodology. It demonstrates the usefulness of the multimethod, multiinformant approach in producing a detailed description, but it also highlights the difficulty in operationalizing the value-laden concepts. First, the treatment planning process is described then it is evalu- ated according to the criteria of the quality checklist.

Describing the Process The role of the Fort Bragg Demonstration case manager in treatment planning is best understood within the context of the treatment team. Case managers acted with the authority of this team. Decisions for starting, continuing, or discontinuing services were made at team meetings in accordance with the clinic’s criteria for level- of-care changes. The team consisted of the treating clinicians, the case manager, the parent or guardian, the child (when old enough to participate), and pro- fessionals from outside the agency such as therapists, therapeutic foster parents, teachers, parole officers, and social service workers. Meetings for establishing or revising the treatment plan occurred frequently, dic- tated by the intensity of current services. Reviews were held no less frequently than every 14 days for hos- pitalized children and no less frequently than every 60 days for children in less restrictive treatment settings. The meetings followed a preset agenda in which the case manager reviewed the child’s history, current services, the child’s progress, and then presented a plan for con- tinued care. Afterwards, the case manager prepared a written treatment plan to reflect the decisions made in the meeting.

Comparison to the Quality Checklist How well did the system meet the expectations of the various stakeholders? The evaluation staff used the previously described checklist (Figure 2) which included three criteria for treatment planning:

Families and Surrogate Families are Full Participants in Planning. Ninety-three per cent of parents reported that they had attended at least one team meeting within the past two months. The chart audit corroborated this: indicating that for 87% of clients, a family member or the client attended at least one of the team meetings held during the 2-month period. More than 90% of

parents responded that they usually or always asked the questions and made the comments that they wanted to at the meetings. Eighty-seven per cent of parents believed that their opinions were usually or always important to the staff members. Ninety-one per cent stated that their input was included in the treatment plan, and 89% stated that they were asked if they agreed with the plan. The expert consultant determined that in 9 1% of the charts there was evidence of active participation by family members or of substantive efforts by staff to include the family. There was also a positive rating on the CPPS-C subscale for family involvement in planning (a group mean of 16.8 out of a possible 20). However, a small group of parents felt marginally involved. Thirteen per cent of parents stated their opinions were valued only sometimes. Nine per cent stated that they had no input in the treatment plan, and 7% said they had not been asked if they agreed with the plan.

The Plan is Comprehensive and Individualized. Children and family members were included in the planning process to assure that an appropriate treatment plan would be developed. Thirty-six per cent of parents thought that the plan met all their child’s needs and 51% reported that the treatment plan met most of their child’s needs. Seventy-four per cent thought it met most or all of their family’s needs. The consultant noted that 54% of the treatment plans addressed all current relevant problems.

In over 90% of the charts, the consultant agreed that the selected services were consistent with the diagnosis and that the planned length and frequency of treatment were appropriate. He also determined that in over 90% of cases, the child received a sufficiently thorough assessment if a level-of-care change occurred or was considered and that the results were adequately reviewed by the staff.

Services are Provided in the Least Restrictive. Most Nor- mative Environment that is Clinically Appropriate. The mean rating of the staff on this CPPS-C subscale was 16.6 out of 20, indicating a positive orientation of program staff toward providing care in the least restrictive environment. The consultant agreed that for 94% of the children, the least restrictive therapeutic environment had been selected, that in 97% of the charts, the level of care met the clinic’s selection criteria for more restrictive settings, and that 97% of the separations of the child from the family (i.e. hospitalization or residential treatment) were appropriate.

DISCUSSION

This study produced a detailed description of case management at the Fort Bragg Demonstration. The evalu-

Evaluating Case Management 127

ation process was particularly useful in defining the planning, coordination, and monitoring activities. According to the project contract, these activities were in fact the main responsibilities of the case managers. Thus, the evaluators were able to examine the aspects of service that were of greatest interest to the project planners.

It was not as useful, however, in documenting counseling and advocacy activities. The activity log did not adequately capture these types of activities. Discussion with case managers revealed that counseling and advocacy activities often occurred in the context of other activities. For example, advocacy might occur in a treatment planning meeting, or counseling might be provided during contact with a parent for other purposes. These activities might not be reflected in the logs. This difficulty was acknowledged in Rothman’s (199 1) sche- matic, in which counseling and advocacy were placed outside the temporal flow of case management activities, indicating that they could occur many times within other processes. As counseling and advocacy activities may be more prominent in some models of case management, methods are needed to record them accurately. A possible solution would be to record multiple activities in the log, although it would be important to avoid creating a cumbersome instrument that decreased com- pliance with data collection.

Even though there were limitations to describing counseling and advocacy activities, the study did permit evaluators to determine how well the implementation matched the ideal system for planning, coordination, and monitoring. In addition, the description contained enough detail that replication of the model and descriptive comparisons to other programs would be feasible. The checklist of quality indicators, generated through concept mapping with various stakeholder groups (Marquart et al., 1992; Trochim, 1989) would provide a tool for making such comparisons. Thus, the evaluation addressed three of the goals Brekke and Test (1992) set out for program descriptions. The fourth, the study of the “critical ingredients” of a model (p. 228) could not be attempted for two reasons.

First, the study design did not permit the evaluation team to assess the unique contribution of case managers to the system of care. While the activities of case managers and other staff members were examined separ- ately, it was not possible to cleanly dissect their respective contributions. For example, in case management systems, the case manager may logically be expected to assure that the care plan is appropriate and that overall coordination of services occurs. However, at the Fort Bragg Demonstration, the entire multidisciplinary team helped assure that this occurred.

The second reason for not attempting to examine the “critical ingredients” was an inability to adequately define the value-laden concepts of case management.

The study of treatment planning illustrates the com- plexity of operationalizing these concepts. A high-quality treatment plan can glibly be described as comprehensive, individualized, and reflecting the needs of the child and family. However, operationalizing these terms is difficult. What is comprehensiveness and can it be measured in an objective and reliable manner? Should a comprehensive mental health plan address all family members’ mental health needs, or is it a plan that addresses all aspects of a child’s life, physical as well as mental health, educational and recreational needs, etc.? Is it all of these things? The magnitude of the task of defining these qualitative terms is immediately appar- ent. Value-laden words are used to describe all components of case management, not just treatment planning. However, until researchers can define and measure these attributes objectively, it will be difficult to judge their importance in case management outcomes.

In the absence of recognized objective criteria, the evaluators chose an expert reviewer to judge the quality of the treatment plan. While this strengthened the assessment by providing the perspective of an experi- enced observer, it did not eliminate the subjectivity of the task. Furthermore, because the reviewer was a mental health provider, a provider’s perspective and values were imposed.

The multimethod, multiinformant approach was used to validate judgments of quality by obtaining different perspectives. However, different perspectives may resist integration. Thus, the need to identify and justify the value base of any evaluation of case management is inescapable.

SUMMARY

With pressure from providers and families who want to know about effective interventions, from policy makers who need answers for program planning, and from foundation boards who want to know if their dollars are well spent, it is understandable that evaluators may attempt prematurely to answer questions about the efficacy of case management interventions. However, as pointed out by Burns et al. (1995), descriptive research is needed to identify the basic structures and processes in case management, before moving on to assessment of outcomes. Evaluations of case management will ben- efit if researchers use a commonly accepted methodological approach. The principles considered in this paper are offered as a starting point for the successful study of case management models, but multiple tasks remain. Attributes of high-quality case management must be operationalized, including those relevant to advocacy and counseling activities. Data collection instruments must be refined for comparison of case management interventions to customary case. Methods


for appropriately integrating quality ratings from multiple informants must be developed.

REFERENCES

ARMSTRONG, MI., & EVANS, M.E. (1992). Three intensive community-based programs for children and youth with serious emotional disturbance and their families. Journal of Child and Family Studies, I, 61-74.

BELCHER, J.R. (1993). The trade-offs of developing a case management model for chronically mentally ill people. Health and Social Work, 18, 20-3 1.

BERKOWITZ, G., HALFON, N., & KLEE, L. (1992). Improving access to health care: case management for vulnerable children. Social Work in Health Care, 17, 101-123.

BICKMAN, L. (1987). The functions of program theory. New Direc- tions,for Program Evaluation, 33, 518.

BICKMAN, L. (1996). The application of program theory to the evaluation of a managed mental health care system. Evaluation and Program Planning, 19(2), 111-l 19.

BICKMAN, L., & PETERSON, K.A. (1990). Using program theory to describe and measure program quality. In L. Bickman (Ed.), New Directions for Program Evaluation (Vol. 47). San Francisco: Jossey- Bass, Inc.

BICKMAN, L., BRYANT, D.M., & SUMMERFELT, W.T. (1993). Final Report of the Quality Study of the Fort Bragg Evaluation Project. Center for Mental Health Policy. Nashville, TN.

BICKMAN, L., HEFLINGER, C.A., PION, G., & BEHAR, L. (1992). Evaluation planning for an innovative children’s mental health system. Clinical Psychology Review, I2, 853-865.

BRANNAN, A.M., SONNICHSEN, S.E., & HEFLINGER, CA. (1996). Measuring satisfaction with children’s mental health services: Validity and reliability of the satisfaction scales. Evaluation and Pro- gram Planning, 19(2), 131-141.

BREKKE, J., & TEST, M.A. (1992). A model for measuring the implementation of community support programs: results from three sites. Community Mental Health Journal, 28, 2277247.

BURNS, B.J., GWALTNEY, E.A., & BISHOP, G.K. (1995). Case management research: issues and directions. In B.J. Friesen, & J. Poertner (Eds.), From case management to service coordination ,for children with emotional, behavioral, or mental disorders: Building on .family strengths. Baltimore: Paul H. Brookes Publishing Co.

CHEN, H.T., & ROSSI, P. (1983). Evaluating with sense: the theory- driven approach. Evaluation Review, 7, 283-302.

CLARK, K.A., LANDIS, D., & FISHER, G. (1990). The relation- ship of client characteristics to case management service provision, Evaluation and Program Planning, 13, 221-229.

DECHILLO, N., STUNTZNER-GIBSON, D., FRIESEN, B., PAULSON, R., & KOREN, P. (1993). Service,fit ,for children and families in the Oregon Partners Project. Regional Research Institute for Human Services, Portland.

GRAHAM, K., & BIRCHMORE-TIMNEY, C. (1989). The problem of replicability in program evaluation: the component solution using the example of case management. Evaluation and Program Planning. 12, 1799187.

HEFLINGER, C.A. (1993). Final report of the implementation study of the Fort Bragg Evaluation Project. Center for Mental Health Policy, Vanderbilt University, Nashville, TN.

HOLLOWAY, F. (1991). Case management for the mentally ill: look- ing at the evidence. International Journul of Social Psychiatry. 37, 2-- 13.

INTAGLIATA, J. (1982). Improving the quality of community care for the chronically mentally disabled: the role of case management. Schizophrenia Bulletin, 8, 6555674.

JERRELL. J., & HARGREAVES, W. (1991). The operating philosophy, of community programs. Working Paper Series 18. Institute for Mental Health Services Research, Berkeley.

KIRKHART, K.E., & RUFFOLO, M.C. (1993). Value bases of case management evaluation. Evaluation and Program Planning. 16, 55 65.

MARCENKO. M.O., & SMITH, L.K. (1992). The impact of a family-centered case management approach. Social Work in Health Care, 17,877100.

MARQUART. J.M., POLLAK, L.B., & BICKMAN, L. (1992). Quality in intake assessment und case management: Perspectives ot administrators, clinicians und consumers. Vanderbilt University. Center for Mental Health Policy, Nashville, TN.

MELVILLE, C., KIBER, J., & HADDLE, H. (1977). The bulanced service system: An approach to the delivery of mental health services. Atlanta: Georgia Mental Health Institute.

MOOS. R. (1988). Community-oriented programs environment .scalc manual (2nd edn). Palo Alto: Consulting Psychologists Press.

MORRISSEY, J.P. (1992). An interorganizational network approach to evaluating children’s mental health service systems. In L. Bickman & D. Rog (Eds.), Evaluutirrg mental health services for children, San Francisco: Jossey-Bass.

MORRISSEY. J.P., & BURNS, B. (1990). Assessing coordinatedcarc ,for children and youth with severe emotional disturbances. Chapel Hill: Health Services Center, University of North Carolina.

PETERSON. K.. & BICKMAN, L. (1992). Using program theory in quality assessments of children’s mental health services, In H.-T. Chen & P. Rossi (Eds.), Using theory to improve program and poliqx evaluutions. New York: Greenwood Press,

ROBINSON, Cr., & TOFF-BERGMAN, G. (1989). Choices in tax management: Current knowledge und practice for mental health programs. Washington, D.C.: Mental Health Policy Resource Center.

ROSENHECK. R.. NEALE, M., & GALLUP, P. (1993). Com- munity-oriented mental health care: assessing diversity in clinical practice. P.sychosocial Rehabilitation Journal, 16, 39950.

ROSSI, P.. FREEMAN, H., & WRIGHT, S. (1979). Monitoring program implementation. In P. Rossi. H. Freeman&S. Wright (Eds.) Evaluation: A systematic approach. Newbury Park, CA: Sage.

Evaluating Case Management 129

ROTHMAN, J. (1991). A model of case management: toward empiri- cally based practice. Social Work, 36, 32&328.

RUBIN, A. (1992). Is case management effective for people with serious mental illness? A research review. Health and Social Work, 17, 139-150.

SCHWARTZ, S., GOLDMAN, H.H.,&CHURGIN, S. (1982). Case management for the chronic mentally ill: Models and dimensions. Hospital & Community Psychiatry, 33, lOOGlOO9.

STROUL, B., & FRIEDMAN, R. (1986). A system of carefor severely emotionall~~ disturbed children and youth. Washington, D.C.: CASSP

Technical Assistance Center, Georgetown University Child Devel- opment Center.

TROCHIM, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, l-16.

WEIL, M. (1985a). Key components in providing efficient and effective services. In M. Weil & J. Karls (Eds.), Case management in human service practice. San Francisco: Jossey-Bass.

WEIL, M. (198513). Adapting case management to specific programs. In M. Weil & J. Karls (Eds.), Case management in human service practice. San Francisco: Jossey-Bass.

Documents

Methodology for evaluating mental health case management