American Journal ofOrthopsychiatry, 68(2), April 1998

PROGRAM FIDELITYIN ASSERTIVE COMMUNITY TREATMENT:

Development and Use of a Measure

Gregory B. Teague, Ph.D., Gary R. Bond, Ph.D., Robert E. Drake, M.D., Ph.D.

Assertive community treatment (ACT) is a complex community-based service ap-proach to helping people with severe mental disorders live successfully in thecommunity. Effective replication of the model and research on critical elementsrequire explicit criteria and measurement. A measure of program fidelity to ACTand the results of its application to fifty diverse programs are presented.

Assertive community treatment (ACT)J \ (Test & Stein, 1976; Test, 1992) haslong been recommended as an effectivetreatment for people with severe mental dis-orders (Drake & Burns, 1995). Since itsoriginal development in Madison, Wiscon-sin as Training in Community Living, a com-munity-based alternative to psychiatric hos-pital care (Stein & Test, 1980), the programhas been widely emulated and frequentlyevaluated (Burns & Santos, 1995; Olfson,1990; Thompson, Griffith, & Leaf, 1990).The general model has been adapted andevaluated in various settings and with vari-ous subgroups of people with severe men-tal disorders, including young adults withrecent-onset schizophrenia (Test, 1992),people with co-occurring alcohol and drugdisorders (Drake, Teague, & Warren, 1990;Teague, Drake, & Ackerson, 1995), veter-ans (Rosenheck, Neale, Leaf, Milstein, &Frisman, 1995), and people who are home-

less (Dixon, Krauss, & Kernan, 1995;Morse, Calsyn, & Allen, 1992), as well aswith families (McFarlane, Stastny, & Dea-kins, 1992).

Several publications have described therationale, important features, and operatingprinciples of ACT (Santos, 1993; Test, 1992;Test & Stein, 1976). However, despite long-standing acclaim and demonstrated effec-tiveness of the model, only recently haveefforts been made to define operationallyand measure empirically its critical dimen-sions (Allness and Knoedler, 1998; Mc-Grew & Bond, 1995; McGrew, Bond, &Dietzen, 1994; Teague et al, 1995). In theabsence of explicit model criteria, explana-tions for variation in outcome must remainspeculative. For proper interpretation ofoutcome findings, an adequate measure offidelity to the ACT model is needed. Thisarticle describes the development and re-sults of using the Dartmouth ACT Scale

Invited for inclusion in this special section of the Journal Work was supported by NIMH grants R01-MH-47567, K2-MH-00839, and K2-MH-00842. Authors are at: Department of Community Mental Health, Louis dela Parte Florida Mental Health Institute, University of South Florida, Tampa (Teague); Department ofPsychology, Indiana University-Purdue University, Indianapolis (Bond); and Department of Psychiatry,Dartmouth Medical School, Lebanon, NH (Drake).

© 1998 American Orthopsychiatric Association, Inc. 216

TEAGUEETAL 217

(DACTS), a product of recent efforts tomake the essential features of ACT opera-tionally explicit.*

MEASUREMENT OF FIDELITY

A number of authors have decried thelack of consistency in providing carefulprogram documentation with reports of theeffectiveness of interventions, includingtreatment (Moncher & Prim, 1991) and ser-vice models (Brekke, 1988; Brekke & Test,1992). Without detailed descriptions of in-terventions, replication is difficult; withoutreliable measurement of interventions, con-clusions about presence or absence of ef-fects are questionable. Careful specificationof the critical components of a model, anduse of measures based on operational defi-nitions of these components provide ameans both to inhibit drift away from themodel and to evaluate the respective con-tributions of theoretically distinct compo-nents (Bickman, 1987, 1990; Bond, 1991;Chen, 1990; Finney & Moos, 1989; Lipsey,1990). Program measurement using multi-ple variables and scales also permits empir-ical decomposition of programs as a sup-plement to theory-driven analyses (Scott &Sechrest, 1989). Empirical measures of pro-gram implementation can be used in sev-eral ways: to shape program performance informative stages; to serve as measures ofdosage or treatment strength; or to define in-clusion criteria for multisite studies (Teagueetal, 1995).

Several approaches have been used todevelop quantitative measures of programimplementation or fidelity of services forpeople with severe mental illness. Brekke(1987) reported use of staff time-logs in-corporating categories of activity that werespecifically matched to critical aspects ofthe intervention. Jerrell and Hargreaves(1991) developed the Community ProgramPhilosophy Scale to measure a wide rangeof community support programs. This in-strument uses staff ratings of a program's

values and practices on 20 four-item scalesreflecting relevant dimensions of commu-nity support philosophy. Brekke and Test(1987) used a battery of existing instru-ments for empirical discrimination of pro-grams.

More recently, and more specifically re-lated to ACT, McGrew et al. (1994) sur-veyed experts to identify critical ingredi-ents of this model. Using the identified in-gredients, they then developed a programsurvey to elicit data on a selected set ofprogram characteristics and performance.Fidelity variables were grouped in three apriori dimensions: staffing, organization, andservice. Program and client hospitalizationdata were collected from a number of ACTprograms in the study. McGrew and col-leagues were then able to demonstrate thatprograms with greater fidelity to the ACTmodel, as defined in terms of the subset ofidentified critical ingredients, were alsomore effective in reducing hospital use.

Teague et al. (1995) drew from the liter-ature on ACT (Bond, 1991; Santos, 1993;Test, 1992; Test & Stein, 1976) to identify13 key dimensions of program implemen-tation in the study of an ACT-like interven-tion for people with severe mental disor-ders and co-occurring substance use disor-ders. Dimensions reflected features both ofthe general ACT model and of a particularmodel of integrated treatment for the twosets of disorders (Drake et al, 1990). Theauthors defined one or more indicators foreach dimension to map onto an anchoredscale, using multiple methods and datasources. Data on outcomes were not thenavailable, but it was evident that differ-ences in program fidelity across experi-mental and control conditions were mostpronounced for those programmatic fea-tures that were the most structurally con-strained, e.g., caseload size and composi-tion. Greater model drift had occurred formore discretionary features, such as overalltreatment approach and in vivo services.

*Copies of the full scale and instructions are available from the first author.

218 MEASURE OF FIDELITY TO ACT

The DACTS, reported here extends thegeneral methods of this approach (Teagueetal., 1995).

Construction of Dartmouth ACT ScaleTwenty-six program criteria for fidelity

to ACT were derived from multiple sources,including literature describing the model(Santos, 1993; Test, 1992; Test & Stein,1976), results from previous work on fi-delity (McGrew et ai, 1994; Teague et al,1995), and expert opinion (McGrew & Bond,1995). The majority of criteria and anchorswere adapted from variables used as indi-cators for the 13 dimensions reported inTeague et al. (1995). Additional variableswere designed on the basis of results re-ported in McGrew et al. (1994). As in Mc-Grew et al., criteria were grouped for con-venience in three a priori dimensions, al-though these dimensions would not neces-

sarily be reflected in the empirical struc-ture. Colleagues studying ACT or relatedprograms provided helpful suggestions forrefinement of items (T. Acker son, D. All-ness, W. Breakey, B. Burns, R. Calsyn, P.Deci, L. Dixon, S. Essock, N. Kontos, A. Leh-man, M. Neale, R. Rosenheck, & A. San-tos, personal communication, July-August,1995). Later editorial refinements for theversion shown in this article were sug-gested in discussion with site evaluatorsfrom the ACCESS program (Randolph,1995).

TABLE 1 identifies and lists descriptorsfor the 26 program criterion variablesgrouped in the three dimensions. (Twovariables, HI 1 and S10, added after thestudy, are also included for a total of 28 cri-teria.) Overall, the current version of theDACTS focuses primarily on structural as-pects of the ACT model, for which behav-

Table 1

PROGRAM CRITERIA FOR FIDELITY TO ACT

HUMAN RESOURCES STRUCTURE/ COMPOSITIONH1 Small Caseload client/provider ratio of 101.H2 Team Approach provider group functions as team

rather than individual practitioners; clinicians know &work with all clients

H3 Program Meeting, program meets frequently to plan,' review services for each client

H4. Practicing Team Leader: supervisor of front-line clini-cians provides direct services.

H5. Continuity of Staffing' program maintains same staff-ing over time.

H6 Staff Capacity: program operates at full staffing.H7 Psychiatnst on Staff, at least one full-time psychia-

trist per 100 clients assigned to programHe Nurse on Staff >2 full-time nurses per 100 clients.H9 Substance Abuse Specialist on Staff: >2 staff with 1

yr training/clinical exp. in substance abuse treatmentH10 Vocational Specialist on Staff. >1 staff member

with >1 yr training/exp in vocational rehab/support.H11 Program Size: sufficient absolute size to provide

consistently the necessary staff diversity & coverage(Data on this vanable not collected in current study)

ORGANIZATIONAL BOUNDARIES01 Explicit Admission Criteria: clearly identified mission

to serve particular population, measurable, operation-ally defined criteria to screen out inappropriate referrals.

02 Intake Rate, takes clients in at a low rate to maintaina stable service environment

03 Full Responsibility for Treatment Services, as well ascase management/psychiatric services, program di-rectly provides counseling/psychotherapy, housing sup-port, substance abuse, employment, S rehab services

04 Responsibility for Crisis Services 24-hour coverageof psychiatric cnses

05 Responsibility for Hospital Admissions program isinvolved in hospital admissions

06. Responsibility for Hospital Discharge Planning pro-gram is involved in planning hospital discharges

07 Time-Unlimited Services, program closes no cases,remains point-of-contact for all clients as needed.

NATURE OF SERVICES51 In-Vivo Services program monitors status, develops

community living skills in community rather than office52 No Dropout Policy, program engages/ retains clients

at mutually satisfactory levelS3. Assertive Engagement Mechanisms' uses street out-

reach, plus legal mechanisms (e g , representative pay-ees, probation/parole, OP commitment) as indicated.

54 Intensity of Service high total amount of servicetime, as needed.

55 Frequency of Contact high number of service con-tacts, as needed.

56 Work With Support System with or without clientpresent, program provides support/skills for client'ssupport network: family, landlords, employers

57 Individualized Substance Abuse Treatment >1 pro-gram member provides direct treatment & substanceabuse treatment for clients w/substance use disorders

58. Dual Disorder Treatment Groups group modalitiesused as tx strategy for people w/substance disorders

59. Dual Disorders Model: uses a stage-wise treatmentmodel that is nonconfrontational, follows behavioralprinciples, considers interactions of mental illness &substance abuse, S has gradual expectations of absti-nence

S10. Role of Consumers on Treatment Team: clients in-volved as team members providing direct services.(Data on this vanable not collected in current study.)

TEAGUEETAL 219

ioral anchors may be more readily devel-oped, rather than on more clinical aspectsthat may be equally important to effective-ness but are harder to measure. For exam-ple, although individualization of treatmentis an important feature of ACT, it is quitedifficult to operationalize adequately, andno such item is currently included.

Structure and content. As can be seen inTABLE 1, the first dimension, human re-sources, addresses composition and struc-ture of program staffing: ratio of clients tostaff; team functioning, including caseloadsharing, regular meetings, and leadershipby a practicing clinician; and specific disci-plines or specialized staffing, includingphysician, nurse, and substance abuse andvocational specialists. Although full andcontinuous staffing are not necessarily lim-ited to the ACT model, they are specifiedbecause failure to achieve them seriouslycompromises program functioning.

The second dimension, organizationalboundaries, addresses program responsibil-ity and relationships with other program-matic components. Criteria include speci-ficity of admission criteria; rate at whichthe program takes on new clients; responsi-bility for specific types of rehabilitativeand crisis services; movement in and out ofhospitals; and commitment to unlimitedservices, as needed.

The third dimension, nature of services,addresses the range and nature of servicesand overall treatment approach. Criteriaspecify services in the community; out-reach and follow-up to maintain clients inservice; explicit use of assertive engage-ment techniques; intensive and frequentservices, as needed; staff working withclients' support systems; and treatment forsubstance abuse problems as needed, on anindividualized basis and through a non-confrontational, stage-wise approach usinggroup modalities.

Item anchors and scoring. Two generalsteps were followed in operationalizingeach criterion. First, an indicator was iden-tified and defined in terms of the nature

and source of data. In order to satisfy agiven criterion, empirical evidence from aparticular source would need a certainvalue. Selection of indicators was guided,in part, by feasibility of definition andmeasurement. When use of quantitative in-dicators to represent a particular criterionwould typically be infeasible, qualitativeindicators were specified. Most criteria,such as frequency of team meetings, haveindicators that fully represent the construct.A few criteria were operationalized lesscompletely. For example, provision of in vivoservices is measured simply by the propor-tion of time that staff spend providing ser-vices out of the office. While this distinc-tion does not fully capture the construct ofspending time with clients in the placesmost relevant for them, it defines a perti-nent and easily measurable aspect of thatconstruct.

The second operationalizing step en-tailed specifying anchors for each point onthe rating scale. Five points were selectedto provide an efficient number of discretepoints that would still approximate a con-tinuous scale. The most critical points fordefinition were the high and low ones. Acentral function of a fidelity measure likethis is to distinguish programs conformingto the ideal from those conforming tonorms for the corresponding conventionaltreatment. Accordingly, because the Madi-son PACT team has long been consideredthe paradigm of ACT, high anchors werederived as much as possible from pub-lished norms for this program (Brekke &Test, 1987). Others were based on descrip-tors in published program literature (Test,1992; Santos, 1993), on a survey of ACTexperts (McGrew & Bond, 1995), and onauthor consensus. Because the context forACT programs—and for alternative treat-ment for clients—is typically services-as-usual, not the total absence of services, lowanchors for quantitative variables were setto an estimated minimal value of typicalcase-management-based services, ratherthan to an extreme value, such as zero.

220 MEASURE OF FIDELITY TO ACT

Thus, the low anchor for annual caseloadretention is defined as 50%. Similarly,scores on the item for small caseload canrange from a high of five for a clienfcstaffratio often or less, to a low of one for anyratio higher than 80. Intermediate pointswere interpolated and adjusted to obtaindistribution across the scale. Overall, thisstrategy was intended to ensure full use ofthe scale and high variance in cross-modelmeasurement.

Programs ratings are generated by map-ping program data or features onto anchorsfor each item. TABLE 2 illustrates anchorsfor three items, one from each a priori di-mension. For quantitative items, programdata or results of specified calculationsmap directly onto ranges specified in an-chors. On item SI in TABLE 2, for example,a program providing 50% of its direct ser-vices in the community would be rated at3. For qualitative items, program featuresare compared with anchors, and the valuecorresponding to the best fit is assigned.On item 04 in TABLE 2, for example, a pro-gram that provides only consultation viatelephone for emergency services would berated at 3.

Data sources. The DACTS is designedfor use by informed raters. Information tobe used in making ratings ideally comesfrom a range of sources, as in the ante-cedent measures (McGrew, et al, 1994;Teague etai, 1995). These sources includereports of program behavior from programsupervisors or staff; documents reflectingprogram authority, responsibility, policies,and procedures; management informationsystem or other sources of quantitative dataon staffing, clientele, and services. Instruc-tions for using the scale include recom-mendations regarding the different sourcesthat are relevant for different items. Whenthe primary rater lacks direct access to allthe data required, a composite rating can bederived from structured interviews withmultiple informants who have direct accessto some of the data. This approach wasadopted in one of the sets of studies re-ported here, where raters were initially lessinformed about program details than wereraters in other studies. Informant categorieswere specified in order to provide the nec-essary coverage of content (R. Calsyn, H.Goldman; personal communication, No-vember 1995).

ITEMH3.Program Meeting.Program meetsfrequently to plan& review servicesfor each client

04.Responsibility forCrisis Services'Program has 24-hr responsibilityfor psychiatriccrises

S1In-Vivo ServicesProgram worksto monitor statusand develop liv-ing skills in com-munity rather thanin office

Table 2

ANCHORS FOR THREE SAMPLE ITEMS

1

Service planningfor each clientusually occursonce a month orless frequently

No responsibilityfor handlingcrises after hours

Less than 20%time in commu-nity

2

At least twice amonth but lessoften than once aweek

Emergency ser-vice has pro-gram-generatedprotocol forclients

2 0 % - 3 9 %

SCALE

3

At least once aweek but less of-ten than twice aweek

Program avail-able by phone,largely in consult-ing role

4 0 % - 5 9 %

4

At least twice aweek but less of-ten than 4 timesweekly

Provides emer-gency servicebackup

6 0 % - 7 9 %

5

At least 4 days aweek & reviewsclient each time,even if briefly

Provides 24-hourcoverage

80% of total ser-vice time in com-munity

TEAGUEETAL 221

METHOD

Study DesignThe research design consisted of exam-

ining four groups of programs imple-mented with varying degrees of intendedfidelity to the ACT model. One group wasintended to be in close replication of theACT model; a second group included in-tensive case management programs basedon ACT principles but with some modifiedelements; a third group incorporated se-lected features of ACT for a specific popu-lation; and a fourth group represented moretraditional case management services. Thefour groups were examined together, em-ploying the "method of known groups"(Cronbach & Meehl, 1955; Pedhazur &Schmelkin, 1991) as an approach to estab-lishing the construct validity of a newscale. This method tests whether an assess-ment approach yields differences corre-sponding to a pre-existing classification. Inthis case, it was expected that the firstgroup would have higher scores on the fi-delity scale than the programs more looselybased on the ACT model, and that these, inturn, would have higher scores than the tra-ditional case management programs.

Program ModelsA total of 50 programs in four groups of

models were evaluated using the DACTS.All programs were designed to serve per-sons with severe mental illness. The firstgroup of programs consisted of 14 differ-ent sites within five separate studies of ef-fectiveness of ACT (Dixon, et al, 1995;Drake et al, 1998; Essock & Kontos, 1995;Primm, 1996; Santos, Deci, Lachance, 1993).Two studies included one site each; two in-cluded two sites each; and one includedseven sites. All were designed to imple-ment as nearly as possible the essential fea-tures of the ACT model and were closelysupervised by researchers, albeit with vary-ing degrees of formal authority over pro-gram implementation. The study incorpo-rating seven sites was designed to evaluatethe addition of integrated treatment for co-

occurring substance use disorders to thebasic ACT model (Drake, et al, 1998, thisissue; Teague, et al., 1995). The studies inthis group began during 1989-1990, hadfollow-up periods ranging from 18 monthsto three years, and conducted final follow-up interviews for the cohorts during 1992-1995. Program size ranged from 40 to 75clients, and caseload ratios ranged fromseven to 13 clients per clinician.

The second group of programs consistedof ten sites from the VA Intensive Psychi-atric Community Care (IPCC) program, anACT-like intensive case management pro-gram implemented in the Department ofVeteran Affairs for persons with severemental illness who had been high users ofhospital services (Rosenheck & Neale, inpress; Rosenheck, Neale, Leaf, Milstein, &Frisman, 1995). Six of the ten sites weregeneral medical and surgical hospitals,four were neuropsychiatric hospitals; allwere implemented and evaluated duringthe period 1987-1995. The model was sim-ilar to the Madison PACT model in severalrespects: it was explicitly designed to pro-vide the clinically indicated degree of ser-vice intensity for each patient; programsize was approximately 45, and caseload ra-tios ranged from 7:1 to 15:1; services wereprovided as much as possible in commu-nity settings; emphasis was placed on prac-tical problem solving and skill buildingthrough a broad range of rehabilitative ser-vices; and continuity of care was assuredby assertive maintenance of contact withand responsibility for each patient. IPCCwas different from PACT in its lesser em-phasis on a team focus, greater staff depen-dence on other programs, and greater pa-tient involvement in other service programs.

The third group consisted of 15 of the 18sites in the Access to Community Care andEffective Services and Supports (ACCESS)program, sponsored by the Center for Men-tal Health Services (Randolph, 1995; Ro-senheck et al., in press). The goal of AC-CESS is to evaluate the impact of increasesin system integration on homeless persons

222 MEASURE OF FIDELITY TO ACT

with severe mental illness. The nine pairsof sites were funded in the fall of 1993, andservice programs were established duringthe first year. Individual service programswere not intended to be replications ofACT, but they shared many of its features,including multidisciplinary team organiza-tion, assertive treatment approaches, andextensive service responsibility. Because ofthese similarities, evaluators of this nation-al-level intervention adopted the DACTSas a measure to compare implementationacross sites. Intentional departures from theusual ACT model included an explicit ex-pectation of time-limited client length-of-stay and, in many cases, structural separa-tion of outreach functions. Each site wasexpected to engage and serve a new cohortof 100 clients every year.

The fourth group consisted of 11 of theprograms that served as control sites for theACT studies in the first group. This groupused standard case management models incomprehensive community mental healthtreatment settings, and had been part of on-going service programs for many years;caseload ratios ranged from 25 to 60 clientsper clinician. Control group data from thetwo studies with one site each were notavailable at the time of analysis, and one ofthe two-site control groups used only onecontrol site. Seven of the programs wereco-located with their corresponding experi-mental ACT programs.

DataCollection. All programs were rated, us-

ing the DACTS, between August 1995 andJanuary 1996. The measure was distributedto principal investigators in the ACT andVAIPCC research studies, along with guide-lines specifying data sources and potentialissues in definition and interpretation. For-mal study periods for the VA and some ofthe ACT programs had recently concluded,so ratings of these and corresponding con-trol programs were made retrospectively.However, each research team was very fa-miliar with its programs on the basis of ex-

tensive data collection and observation ofboth experimental and control groups, andprincipal investigators had reacted to ear-lier drafts of the DACTS. Final ratings ofthe VA IPCC programs were made by thecentral research team on the basis of exten-sive data collection and observation, aswell as provisional ratings and other infor-mation from site managers. Ratings forACCESS sites were made by lead evalua-tors within each state, using preliminaryratings based on structured interviews withselected informants from each site. Infor-mants varied in their familiarity with thecontent of specific items so, in making fi-nal composite ratings, raters weighted theirresponses accordingly. ACCESS serviceprograms had been fully implemented forat least one year; thus, all the programs inthe current study had matured at least be-yond the initial start-up phase. Data for allprograms except ACCESS were sent to thefirst author on paper forms.

Analysis. Descriptive statistics were gen-erated for each of the variables to evaluatethe distribution of scores. Data from theVA IPCC programs did not include scoreson substance abuse treatment variables, soanalyses including these variables wereperformed only on the other three groups.Because multiple ratings were available foronly one set of programs, no attempt wasmade to assess agreement across data sources.Principal components analysis with vari-max rotation was used to identify the struc-ture of variation in this dataset; althoughthe ratio of cases to variables was approxi-mately 2:1, the method is appropriate forthis kind of descriptive purpose. Variablesloading primarily on each factor were aver-aged to create scale scores. Correlationsamong scales were calculated, and Cron-bach's alpha coefficients were calculatedfor scales, for the three a priori dimensions,and for the overall scale to assess internalconsistency. Program groups were com-pared on all scales using analysis of vari-ance and Tukey HSD tests with alpha set to.05, a relatively liberal criterion appropri-

TEAGUEETAL 223

ate for exploratory purposes (Huberty &Morris, 1989). Hierarchical, agglomerativecluster analysis, employing the centroidmethod for calculating linkage distance,was performed on all programs, using fac-tor scores as variables. Resulting clusterswere evaluated by the authors and con-tributing researchers on the criterion offace validity, based on familiarity with pro-gram characteristics. All analyses wereconducted using SPSS (Norusis, 1994).

RESULTS

Item ScoresItem distributions were examined to de-

termine their utility in this population ofprograms. TABLE 3 shows means, standarddeviations, skewness, and number of ratedprograms for the 26 items used in thisstudy. The majority of items showed ap-propriate distributions for the sample. Allbut two items used the full range of the 5-point scale; the two exceptions were case-load size and staffing capacity, for whichno programs were rated at the lowest valueof 1. All means but three were between 2and 4; the exceptions were that most of theprograms maintained close to full staffingcapacity, had a low intake during the pe-riod rated, and did not have their own vo-cational staffing. All standard deviations

but two were greater than 1; the exceptionswere lower variance on staffing capacityand on the dual diagnosis model. Aggre-gated across all items, percentages of val-ues assigned on the 5-point scale were asfollows: 1=13%, 2=17%, 3=23%, 4=29%,5=18%.

To measure the degree to which theoverall measure tapped an empirically co-herent construct of ACT, Cronbach's alphawas calculated for all 26 items. As shownin the overall scales section of TABLE 4, thevalue for the 33 programs reporting allitems was .92. Seven items had item-totalcorrelations of approximately .4 or less.When these were removed, the resultingvalue of Cronbach's alpha for the 19 re-maining items increased to .94. The 19items are identified on TABLE 3. However, itis not evident that ACT is necessarily anintrinsically single construct. Because theintent of the current study was to explorevariation, if any, beyond a single construct,and because each item was designed to tapan important and substantive aspect of theservice model, all variables were includedin subsequent analyses.

Factors and ScalesThree a priori scales were calculated as

the mean of included items. TABLE 4 shows

Table 3

MEANS AND DISTRIBUTIONS OF ITEM SCORES

VARIABLE M SD SK"

Human ResourcesH1 Small caseload (A/=50) 3 88 1.02 -0.70H2. Team approach (A/=50) 2.68 1.42 0.29H3. Team meeting (N=49) 3.88 1 09 -0.65H4. Team leader rote (W=50)b 3.82 1.32 -0.87H5. Staffing continuity (W=50)b 3.98 1.02 -1.04H6. Staff capacity (N=50)b 4.26 0 72 -0.78H7. Psychiatrist on staff (AA=50) 3.24 1.39 0.02H8. Nurse on staff (W=50) 3.52 1.53 -0.38H9. Subst. abuse spec. (W=50)b 2.66 1.73 0.40H10.Vocational spec. (A/=50)b 1.78 1.22 1.64

Organizational Boundaries01 . Admission critena (/v=50) 3 90 1.18 -0.8802. Intake rate (W=50) 4.32 1.06 -1.9803. Tx responsibility (A/=49) 3 33 1.18 -0.6804. Crisis services (W=50) 2.92 1 38 0.25

VARIABLE

05. Hosp. admiss resp. (W=49)06. Hosp disch. plan. (N=50)07 Unlimited service (W=50)b

Nature of Services51 In-vivo services (W=50)52 No-drop policy (A/=49)53. Assert, engage (N=50)54. Service intensity (A/=50)55. Contact frequency (W=50)56. Work w/supports (W=50)57. Subst. abuse tx (N=35)S8 Subst abuse orps(/v=35)b

S9. Dual disord, model (W=36)Means

26 items (W=50)19 items (W=50)

M3.553.763.36

3.223 633.763 342.823.063.062 862.97

3 383 46

SD1 021.081.50

1 041 091 101.221.121.041.391.330.81

0.690 79

SK«-0 45-0.71-0.65

-0.12-0.70-0.56-0 140 460.33-0.040.120.39

-0.49-0.60

•SK=Skewness.blndicates items not used in 19-item measure.

224 MEASURE OF FIDELITY TO ACT

Table 4

SCALE INTERNAL CONSISTENCIES AND CORRELATIONS BETWEEN

INTERNAL CONSISTENCY

SCALE ITEMSOverall Scales

26-item19-item

A Priori Scales1. Human Resources2 Organization Boundaries3 Services

Empirically Derived Scales1 Team/Intensity2 Community Treatment3 Engagement/Retention4 Substance Abuse Tx5 Specialized Staffing6 Caseload Distribution7 Staffing/Continuity8 Vocational Specialist

2619

1079

56433221

W I

3333

494834

4948493450505050

CHRON a 1

0.920.94

0.770.790.83

0 870 870.770.740 770.640.65—

—0.760.72

—0.730 490440.540 480 160.15

2

—0 88

—0.380 530.560.540.24013

SCALES

SCALE CORRELATIONS

3

—

0.290 400.370 340.07

4 5

—0 37 —0.45 0.43

-0 05 0 180.00 0 22

6

—0 010.04

7

0 52

8

—

Cronbach's alpha for each, as well as cor-relations among them. As expected, thescales are not strongly coherent or differen-tiated. Internal consistency is only moder-ately high, given the number of includedvariables; it is constrained for each scale byone or more variables with low item-to-total correlations. Additionally, interscalecorrelations are relatively high, suggestingthat the variation in this sample is not struc-tured around these three a priori dimen-sions.

Accordingly, principal components anal-ysis was used for exploratory data reduc-tion. Using an eigenvalue limit of 1.0, eight

factors were identified, explaining 78% ofthe variance. Variance explained by the firstunrotated factor, a measure of the strengthof the central construct, was 36%. Varimaxrotation yielded factors with satisfactoryconstruct coherence. Because each of thelast three factors had fewer than three vari-ables with high loadings, other solutionswith smaller numbers of factors were de-rived. Alternative solutions were examinedfor general conceptual consistency withinfactor and discrimination across factors.However, the initial 8-factor solution dem-onstrated the best consistency and discrim-ination overall. Since factor analysis is used

Table 5

FACTORS, PERCENT OF TOTAL VARIANCE, AND PRINCIPAL ITEM

FACTORTeam & Intensity

Contact frequencyTeam meetingTeam approachService intensitySmall caseload

Community TreatmentIn-vivo servicesHospital admission responsibilityTreatment responsibilityAdmission criteriaCnsis servicesWork with supports

Engagement/RetentionNo-drop policyAssertive engagementHospital discharge planningTime-unlimited service

VAR.14 2%

13.5%

10.8%

LOADING

0.820.680.640 640.63

0.680.630 620.610 610.58

0.790.730.690.57

FACTORSubstance Abuse Treatment

Substance abuse groupsSubstance abuse treatmentDual disorder model

Specialist StaffingSubstance abuse specialistNurse on staffPsychiatrist on staff

Caseload DistributionTeam leader roleIntake rate

Staff Capacity & Continuity

Staff capacityContinuity of staffing

Vocational SpecialistVocational specialist

LOADINGS

VAR I10 5%

9 0%

7.4%

7 1%

5 7%

LOADING

0.830.780 72

0.780.740.58

0.840 63

0 900 72

0.87

TEAGUE ET AL 225

here for its heuristic and descriptive value,these are the results presented.

TABLE 5 shows the eight factors and vari-ables, with primary loadings for each fac-tor. Across all scales, these loadings rangedfrom .57 to .90, with a mean of .70. The first,second, third, and fifth factors, together ac-counting for almost half of the variance,covered most of the salient features ofACT. The team/intensity factor includeditems measuring shared caseload and teammeeting frequency, as well as other itemsshown. The community treatment factorincluded a structure and range of servicetypes characteristic of the ACT model; inaddition to other items, this factor coveredrehabilitative and support services pro-vided in the community and a clearly de-fined service population. The engagement/retention factor included a demonstratedcommitment to ongoing services, as well asactions to develop and maintain engage-ment over time. The specialist staffing fac-tor included the presence of the listed disci-plines or forms of expertise on the team.

The substance abuse treatment factor in-cluded three items defining specific as-pects: use of specialized groups, amountof individualized treatment for co-occur-

ring disorders, and use of a staged modeladapted for the population of people withsevere mental disorders.

The remaining three factors each hadfewer than three variables. Of the two inthe caseload distribution factor, the teamleader role was defined as that of an activeservice provider. The two variables in thestaff capacity and continuity factor in-cluded items measuring the degree towhich positions are kept filled, as well aspresence of the same staff over time. Thevocational specialist factor was defined bya single item.

Each factor was converted into a sub-scale by taking the mean of all includeditems, following the usual convention forlinear additive scales (Nunnally, 1978). Asshown in TABLE 5, all variables were in-cluded, each in only one scale. All butthree variables had only minimal loadingson any additional factor. Time-unlimitedtreatment, which loaded primarily on en-gagement/retention, also loaded negativelyon community treatment and positively oncaseload distribution. Psychiatrist on team,which loaded primarily on specialist staff-ing, also loaded on team/intensity. Finally,work with support system, which loaded

Table 6

SCALE SCORES BY PROGRAM TYPE

39 81 .000 A>V, H>C36 74 .000 A>V, H>C

24.7430.13

.000 A>V, H>C

.000 A>V, H>C

PROGRAMS' COMPARISONS

SCALES ALL(/v=50) ACT(AM4) VA(W=10) HMLS(N=15) CTRL(W=11) F(3,46) p DIFF.b

Overall26-item 3 38 (0.69) 4.01 (0.39) 3.52 (0.54) 3.42 (0.22) 2 38 (0.36)19-item 3.46(0.79) 4 17(0 45) 3.51(0.67) 3.60(0.24) 2 31(0.40)

A PrioriHuman Resources 3.40 (0.73) 4.00 (0 44) 3 49 (0.63)Organiz. Bounds 3.59 (0 80) 4.32 (0.46) 3.74 (0.50)Services 3.20 (0.72) 3.79 (0 40) 3.32 (0.60)

Empirically DerivedTeam/Intensity 3.32(0.95) 4 33(0.45) 2 94(0.65) 3 51(0.47) 2.11(0.47)Community Tx 3.33(0.89) 3.81(0.60) 3 38(0 67) 3 84(0 39) 1.97(0.25)Engage/Retention 3.63(0 91) 4.44(0 61) 4 00(0.54) 2 92(0 84) 3.20(0.60)Subst Abuse Txc 2.95(0 99) 3.36(0.66) — — 3.11(0 85) 1.92(0.90)Specialized Staff. 3.14(1.29) 4.02(0 96) 3.57(1.28) 3.04(110) 1.76(0.62)Caseload Distrib. 4.07(103) 4.71(0.47) 4.35(0 67) 4.07(0.68) 3.00(1.38)StaffVContinuity 4.12(0.76) 4.32(0.99) 4 15(0.71) 3.97(0.58) 4 05(0.72)Vocational Spec. 178(1.22) 1.86(1.35) 2.30(189) 167(0 82) 1.36(0.50)

3.41 (0.45) 2.40 (0.34)3.60 (0.50) 2.51 (0.44)3.27 (0.45) 2.26 (0.51) 21.14 .000 A>H, C

V, H>C

42.5836.7014.928.05

10.969110.561 11

.000 A>H>V>C

.000 H, A, V>C

.000 A, V>C, H

.001

.000

.000NSNS

A, H>CA, V, H>CA, V, H>C

•Mean scores are given, with standard deviations in parentheses."Tukey's HSD test (.05)'Overall N for Substance Abuse Treatment Scale=37.

226 MEASURE OF FIDELITY TO ACT

primarily on community treatment, alsoloaded on vocational specialist.

TABLE 4 shows Cronbach's alpha foreach empirically derived scale as well ascorrelations among them. The first fivescales showed satisfactory internal consis-tency, with scores of .74 or higher. Thenext two, both 2-item scales, had lower butusable scores of .64 and .65. Correlationsbetween all pairs of scales except one were.56 or less (i.e., shared variance of 31% orless), typically much less, with a mean cor-relation among all scales of .32. Team/in-tensity and community treatment weremost strongly correlated with other scalesand especially with each other.

Comparison Across Program TypesThe four different program types were

compared on all scales. TABLE 6 showsscale means for the whole sample and foreach program type. Overall fidelity-to-ACTscores, based both on the total 26 items andon the 19 items with item-total correlationsabove .4, are shown in the first section. Re-spective values using the two scales variedslightly, with the leaner 19-item scaleyielding more extreme scores, but the re-sults were equivalent. ACT research sitesshowed higher overall fidelity scores thanall other programs, and control programsshowed the lowest. Homeless and VA pro-gram types were not significantly differentfrom each other on these scales.

On the a priori scales shown in TABLE 6,the difference between ACT and VA pro-grams on the services scale was just mar-ginally nonsignificant with the statisticaltest used. Otherwise, consistent with theirhigh intercorrelations, these scale scoresfollowed the same pattern across programtypes as the overall fidelity scales.

Six of the eight empirically derivedscales (see TABLE 6) showed significantbetween-type variance. The first, team/in-tensity, sharply discriminated among allfour program types. The next five scalesdiscriminated controls from ACT but var-ied in the way they discriminated among

other program types. The VA programs didnot differ from ACT on these scales exceptfor substance abuse treatment, on whichV A programs were not rated. The homelessprograms were similar to ACT except onengagement/retention, on which they wereequivalent to controls. In general, all pro-gram types had low involvement of voca-tional specialists and high staffing/continu-ity, and did not differ significantly on thesetwo scales.

Empirical TypingCluster analysis was performed on the 50

programs, using the eight factor scores thathad accounted for approximately four-fifthsof the variance among the original set of 26variables. Two cluster solutions had gener-ally good face validity, one with ten clus-ters, the other with four. Cluster assign-ments in each solution were shared and dis-cussed among researchers familiar with theprograms. There was consensus that simi-lar programs were clustered appropriatelyat each level. In many instances, there wassubstantial consistency between cluster as-signments and nominal program type.However, programs that were outliers intheir types seemed to be appropriately clas-sified with other program types. Since de-scription of the ten-cluster solution is be-yond the scope of the present article, con-sideration is limited to the simpler four-cluster solution.

TABLE 7 shows the cross-tabulation oftype and cluster memberships for the four-

Table 7

PROGRAM CLUSTER MEANS ANDMEMBERSHIP BY PROGRAM TYPE

FIDELITY"

OUSTS*" MSARTWASC

Total

3.803.703.392.303.38

SD.45.32.22.24.69

PROGRAM N

ACT VA10 4

1 43 10 1

14 10

HMLS7260

15

CTRL100

1011

TOTAL227

101150

Note. SA=Strong ACT, RT=Rehabilitation Team; WA=Weak ACT; SC=Standard Case Management"26-item Overall Fidelity Scale^Cluster Comparisons; F(3,46)=45.08; p=.000; SA>WA,SC « SA, RT, WA>SC (Tukey's HSD test).

TEAGUE ET AL 227

cluster solution as well as overall fidelityscores on the 26-item DACTS for eachcluster. On the basis of their respective pro-gram type memberships and factor scores,the four clusters were characterized asstrong ACT programs, rehabilitation teams,weak ACT programs, and standard casemanagement (CM) programs. The strongACT cluster was dominated by ACT re-search programs, but about half of the VAand homeless programs were in this group.These programs had especially high scoreson team/intensity, community treatment,specialist staffing, engagement/retention,and staffing/continuity. The rehabilitationteam cluster was high on community treat-ment, substance abuse treatment, specialiststaffing, and vocational specialist; thisgroup included programs that were some-what less intensive and team-oriented, butmore strongly focused on community-based vocational interventions. The thirdcluster was designated weak ACT becauseit included several programs that had ACT-like structures but had generally achievedlower functioning as teams, having beenless successful in hiring and retaining ap-propriate staff, and client turnover hadbeen high; accordingly, they scored loweron staffing/continuity, specialist staffing,and engagement/retention. The fourth group,standard case management, included allcontrol sites except one that had done wellat emulating the ACT intervention, as wellas one VA site that had not been successfulin implementing the intensive case man-agement program.

DISCUSSION

The foregoing sections have presentedthe development and use of a 26-item an-chored measure of fidelity to ACT, theDartmouth ACT Scale (DACTS). Items rep-resent criteria for model fidelity that werederived theoretically or empirically. Speci-fic indicators and data sources for these cri-teria were selected in terms of both sub-stantive relevance and feasibility of mea-surement. Performance of the measure was

evaluated by applying it to 50 programs, in-cluding programs expected to have high fi-delity to ACT, programs with features sim-ilar to those of ACT, and standard treat-ment programs. Issues to be discussed in-clude measurement properties of the scale,variation among programs, potential uses,and possible future development of themeasure.

Measurement PropertiesFindings on the 26 items used to form

the DACTS were generally as expectedwith this set of programs: most variables,as well as the overall scale, showed appro-priate means and distributions. Because allbut the control programs represented ACT-like models, means somewhat above 3.0,the midpoint of the range, are consistentwith the design intent to capture the vari-ance between standard case managementand a fully implemented ACT program.Mean scale values for high and low groupsreflected overall use of the scoring rangewithout risking floor and ceiling effects.

Further evaluation is necessary to deter-mine modifications appropriate to specificitems, e.g., to those defining the last two fac-tors. The vocational specialist variable hada very low mean, was relatively skewed, andwas not well correlated with the centralACT construct in this set of programs. How-ever, this single variable played a strongrole in discriminating a cluster of ACT-likeprograms that were especially vocationallyoriented. Further, as the norms for effectiveinterventions for people with severe mentaldisorders evolve, vocational interventionswill probably be more salient (Bond, Drake,Mueser, & Becker, 1997). Although therewas less variance on staffing capacity andcontinuity items, this scale played a role inidentifying poorly implemented programs.Maintenance of staffing is not a require-ment unique to ACT, but it is fundamentalto continuity of care, a core feature of themodel. These two factors, along with otherfactors defined by small numbers of items,may require supplementation through addi-

228 MEASURE OF FIDELITY TO ACT

tional variables consistent with the con-struct.

Support for validity of the DACTScomes from four types of sources. First, theitems have generally good face and contentvalidity, explicitly reflecting the essentialfeatures reported in an extensive literatureabout the model. Second, although theMadison PACT program was not includedin the sample as a criterion case, trainersfrom that program have given a generallypositive critique of the items and standardsset in the measure (D. Allness, personalcommunication; May, 1996). A similar con-clusion was presented in an authoritativereview of approaches to operationalizingthe model (Meisler, 1997). Third, by usingthe DACTS for groups of programs withgenerally known properties, the currentstudy empirically establishes the measure'scapacity to function as designed, i.e., todistinguish programs at different levels offidelity to ACT. Finally, there is some pre-liminary evidence of predictive validity. Inan unpublished study evaluation of 12newly formed ACT teams, Bond, Salyers,and Fekete (1996) found that scores on theDACTS were associated with reduced hos-pital use and lower levels of staff burnout.In preliminary analyses by the first andthird authors and colleagues, componentsof fidelity as measured on the DACTSwere related to substance abuse outcomesin dual-diagnosis ACT programs. More di-rect and rigorous evaluation of predictivevalidity awaits further research.

The empirical support for validity neces-sarily provides only indirect support for re-liability, which was not directly evaluatedin this study. Work remains to be done inassessing different types of reliability forthe DACTS, including agreement both be-tween data sources and between interview-ers collecting the information. A study us-ing a very similar measurement approachfound that agreement between interviewerswas very high (Bond, Becker, Drake, & Vog-ler, 1997). On the other hand, agreementbetween data sources in recently formed

teams was found to be lower, at least withina narrow range of program variability (Bondetal, 1996; Test, Bond, & McGrew, 1997).

Variation Among ProgramsFactor analysis was used to reveal pat-

terns of relationships among the criteria,i.e., to observe the structure of variation inthe implementation of ACT in the sampleof programs. The eight factors showedgenerally good face validity in that primaryvariables for most of them allowed forclear interpretation. The first three factors,team/intensity, community treatment, andengagement/retention, along with the fifth,specialist staffing, captured most of the coretraditional criteria of ACT. The fourth fac-tor, substance abuse treatment, althoughhistorically less explicitly associated withACT, has come to be recognized as an inte-gral treatment component (Stein & Santos,1998):

As was evident both in factor loadingsand in correlations between resulting scales,there is a close association between team/intensity and community treatment. The for-mer covers both the capacity to providehigh-intensity services as needed and a teamstructure that enhances the staff informa-tion base, allowing optimal deployment ofhuman resources. These characteristics wouldappear to enhance significantly a program'sprospects for addressing the criteria incommunity treatment, which covers therange of responsibility and activity neces-sary to address clients' needs. Engagement/retention includes criteria reflecting theseparable issues of how actively clients areengaged and how long they are retained inthe program. Assertive engagement andtime-unlimited treatment have historicallybeen core concepts for ACT, but in theACT-like programs in this sample therewas greater variation. The ACCESS pro-grams, for example, were mandated to servea new population of 100 homeless clientseach year. Clearly, some form of transitionor graduation is necessary in this context.

Two of the remaining three factors were

TEAGUEETAL 229

represented by only two criteria, and one ofthem by a single criterion. In the normalcourse of instrument construction, such di-mensions would be modified by elimina-tion, elaboration, or integration with otherdimensions. They are reported here, how-ever, because the present purpose is par-tially exploratory and descriptive. Staf-fing/continuity was internally consistent,interpretable, and useful in differentiatingprograms. The caseload distribution factorposed a greater challenge in interpretation,and this label, especially, is necessarily ten-tative. Both of the included criteria werederived from the work of McGrew et al.(1994), in which the role of the team leaderas provider of direct service alongsideother team members was shown to be apredictor of improved outcome. Althoughthe authors did not explicitly address howthis might occur, a possible mechanism isimproved organizational functioning andclimate, which itself could follow frommore egalitarian roles fostered by a leaderwho is involved with treatment and is notjust a distant administrator. A low, con-trolled intake rate has been proposed as acritical ingredient; this, too, would ensurethat the distribution of relationships be-tween staff and clients would be even andcomfortable over time. Unlike most of thefirst five factors, which appear to reflectmore traditionally central features of ACT,these last three seem improbable candi-dates for replication in other studies.

Utility of the DACTSAn important use of a fidelity measure is

to discriminate programs faithful to a mod-el from those that are not. Both overall andindividual scale scores on the DACTSplayed a role in contrasting the differenttypes of programs included in this sample.Control programs scored below all otherprograms, both overall and on six of theeight scales. The VA and homeless pro-grams were not designed to replicate thefull array of ACT characteristics, and theyscored appropriately at intermediate levels

of overall fidelity, significantly differentfrom both ACT and controls but not fromeach other. The team/intensity scale dis-criminated among all four programs, dis-tinguishing the more team-based and inten-sive homeless programs from the VA pro-grams. The homeless programs were notdesigned for long-term retention of clientsand consequently scored on a level withcontrols on the engagement/retention scale,despite commitment to assertive outreach.It should be noted that the criterion for as-sertive engagement was operationalized insuch a way that it emphasized use of legalmechanisms relative to less easily measur-able engagement strategies. Many of thehomeless programs attempted to minimizeuse of legal mechanisms, so their lowerscore on this scale may have been partly anartifact of operationalization.

Although overall and scale scores on theDACTS were useful in discriminatingamong groups of substantially differentprograms, the measure was not designed tomake fine distinctions within sets of verysimilar programs. Anchors for the five-point scales were set to the range betweenconventional services and ACT; sensitivityto differences is lowered when a truncatedrange must be used. The factor structure re-ported here was derived by using a samplecovering most of the range of measure-ment; although factors approximating thefirst five may well emerge in similar sam-ples, the overall structure is not necessarilyreplicable or relevant in more homoge-neous samples. On the other hand, the gen-erally satisfactory classification of pro-grams using cluster analysis, as reportedhere, suggests that thorough use of the totalvariance may have some utility beyond thesimpler use of scale scores. As noted, theimportance of these sources of variation foroutcome remains to be examined. Use ofthe DACTS in studies that include outcomedata will be necessary in order to determineits utility in verifying that putatively criti-cal ingredients are indeed critical.

Aside from characterizing differences

230 MEASURE OF FIDELITY TO ACT

among types of programs and evaluatingrelative contributions to program effective-ness, a fidelity measure such as the DACTShas other important applications. Programimplementers and researchers can use it asa reference for ensuring program qualityeither overall or in terms of each individualcriterion. The overall fidelity score can beused as a general indicator of fidelity, per-haps with a specific threshold for inclusionof sites in a particular study. Greater use ofthis or similar measures would allow com-parison with norms for different types ofprograms; the data reported here may serveas preliminary norms. A number of consul-tants have reported that the format of theDACTS makes it useful as a training in-strument by providing concrete quantita-tive or qualitative targets for self-eval-uation and emulation. This type of measureis also amenable to multiple applications ina given setting. The full set of criteria canbe used for implementation monitoring andquality management, while reduced sets ofvariables such as the more reliable 19-itemoverall fidelity scale or selected subscalescan be used for statistical analysis.

Further DevelopmentThe version of the DACTS reported here

is a step in the process of operationallydefining the critical components of ACT.This evolution is driven by several factors,including more careful explication of exist-ing notions and experience with the model,advances in effectiveness research, andelaboration of program theory. The inclu-sion of substance abuse criteria representsan instance of elaboration, since this clini-cal area had not initially been featured as apart of the model. After collecting the datain this study, two items were added: pro-gram size, and the role of consumers on thetreatment team. The first follows from theexperience of the leading ACT traininggroup from Madison, where absolute teamsize was found to be a critical element inensuring the necessary coverage (D. All-ness, personal communication, May, 1996).

Along with the criterion of time-unlimitedtreatment, the importance of this element iscurrently subject to debate among propo-nents of ACT models, but it seems none-theless an important dimension to track.The second new item, on consumer roles,follows from a gradual increase in attentionin the field to the potential benefits of con-sumers' participation in the direct provi-sion of services. Again, this is an importantfeature to evaluate across programs, even ifit is not currently considered a formal crite-rion.

Additional modifications to the DACTShave been recommended, including moreexplicit elaboration of the clinical servicesthemselves. The authors expect to workclosely with the principals of PACT, Inc.,a project established by the National Al-liance for the Mentally 111 to promulgatethe ACT model throughout the country(Flynn, et al, 1997). PACT, Inc. is ex-pected to provide both training of staff andcertification of programs. This group hasdeveloped a detailed training manual (All-ness & Knoedler, 1998). In this context, theDACTS may serve as a prototype for thekind of multipurpose measure that such anendeavor would require, supporting train-ing, empirically based fidelity evaluation,and multidimensional process measure-ment to complement comprehensive out-come measurement. This complete rangeof applications is necessary for measure-ment of model fidelity in order to evaluatethe quality and quantity of critical dimen-sions in services.

CONCLUSION

ACT is a complex community-based ser-vice approach that has been demonstratedas helpful for people with severe mentaldisorders adapt to lives in the community.Widespread replication of the model hasbeen attempted. However, without very ex-plicit operational criteria for program struc-ture and processes, developers of new pro-grams cannot be sure that they are follow-ing the model accurately. Further, without

TEAGUE ET AL 231

careful measurement of the multiple di-mensions of this complex program model,researchers can neither be sure that specificprograms are true to the model nor identifythe relative contribution of program com-ponents to effectiveness. Pending betterunderstanding of how specific componentsof treatment programs and settings interactwith specific characteristics of the peopleinvolved to produce desired outcomes, re-searchers must sample from a range ofplaces along the hypothesized causal chain,including both structural elements and in-termediate process elements.

The Dartmouth ACT Scale reported hereprovides a preliminary, standardized mea-sure of fidelity to the ACT model. Empiri-cally and theoretically derived criteria ad-dress multiple dimensions of program struc-ture and process. These criteria and the over-all measure discriminated among a varietyof ACT and ACT-like programs and dis-criminated these from conventional treat-ment programs. Although the current mea-sure will be refined to enhance its utility indeveloping and evaluating ACT programs,it has been useful in its current form fortraining and fidelity determination.

REFERENCESAllness, D.J., Knoedler, W.H. (1998). The PACT

Model of Community-Based Treatment for Per-sons with Severe and Persistent Mental Illnesses'A Manual for PACT Start-Up. Arlington, VA:NAMI Anti Stigma Foundation.

Bickman, L., (Ed.). (1987). Using program theory inevaluation. San Francisco: Jossey-Bass.

Bickman, L., (Ed) (1990). Advances in program the-ory. San Francisco: Jossey-Bass.

Bond, G.R., (1991). Variations in an assertive out-reach model. In N.L. Cohen (Ed.), New Directionsfor mental health services (pp. 65-80). San Fran-cisco: Jossey-Bass.

Bond, G.R., Becker, D.R., Drake, R.E., & Vogler,K.M. (1997). A fidelity scale for the IndividualPlacement and Support model of supported em-ployment. Rehabilitation Counseling Bulletin, 40,265-284.

Bond, G.R., Drake, R.E., Mueser, K.T.,Becker, DR.(1997). An update on supported employment forpeople with severe mental illness. Psychiatric Ser-vices, 48,335-346.

Bond, G.R., Salyers, M.P., & Fekete, D.M. (1996).Illinois ACT Project: Final report. Indianapolis:Indiana University-Purdue University.

Brekke, J.S., (1987). The model-guided method ofmonitoring program implementation. EvaluationReview, II, 281-300.

Brekke, J.S. (1988). What do we really know aboutcommunity support programs? Strategies for bettermonitoring. Hospital and Community Psychiatry,39, 946-952.

Brekke. J.S., & Test M.A. (1987). An empirical anal-ysis of services delivered in a model communitysupport program. Psychosocial RehabilitationJournal, 10(4), 51-61.

Brekke, J.S., & Test, M.A. (1992). A model for mea-suring the implementation of community supportprograms: Results from three sites. CommunityMental Health Journal, 28, 227-247.

Burns, B.J., & Santos, A.B. (1995). Assertive com-munity treatment: An update of randomized trials.Psychiatric Services, 46. 669-675.

Chen, H.T. (1990). Theory-driven evaluations. New-bury Park, CA: Sage Publications.

Cronbach, L.J., & Meehl, P.E. (1955). Construct va-lidity in psychological tests. Psychological Bul-letin, 52,281-302.

Dixon, L.B., Krauss, N., & Keman, E. (1995). Modi-fying the PACT model to serve homeless personswith severe mental illness. Psychiatric Services,46, 684-688.

Drake, R.E., & Burns, B.J. (1995). Special section onassertive community treatment: An introduction.Psychiatric Services, 46. 667-668.

Drake, R.E., McHugo, G.J., Clark R.E., Teague,G.B., Xie, H., Miles, K., & Ackerson, T.H. As-sertive community treatment for patients with co-occurring severe mental illness and substance usedisorder: A clinical trial. American Journal ofOr-thopsychiatry, ({5,201-215.

Drake, R.E., Teague, G.B., & Warren, R.S. (1990).New Hampshire's dual diagnosis program for peo-ple with severe mental illness and substance abuse.Addiction and Recovery, 10, 35-39.

Essock, S.M., & Kontos, N. (1995), Implementing as-sertive community treatment teams. PsychiatricServices, 46, 679-683.

Finney, J.W., & Moos, R.H. (1989). Theory andmethod in treatment evaluation. Evaluation andProgram Planning, 12, 307-316.

Flynn, L., Bevilaqua, J., Meisler, N., Knoedler, W.,Allness, D., & Steinwachs, D. (1997). NAMI Anti-Stigma Foundation, NAMI/PACT initiative for na-tional dissemination of the PACT model. Commu-nity Support Network News, 11(4), 10-11. (Centerfor Psychiatric Rehabilitation, Boston University)

Huberty, C.J. & Morris, J.D. (1989). Multivariateanalysis versus multiple univariate analysis. Psy-chological Bulletin, 105, 302-308.

Jerrell, J., & Hargreaves, W. (1991). The operatingphilosophy of community programs: Working pa-per #18. San Francisco: UCSF Institute for MentalHealth Services.

Lipsey, M.W. (1990). Theory as method: Small theo-ries of treatments. In L. Sechrest, E. Perrin, & J. Bun-ker (Eds.), Research methodology: Strengtheningcausal interpretations of nonexperimental data(AHCPR Conference Proceedings). Washington,DC: US Department of Health and Human Services.

232 MEASURE OF FIDELITY TO ACT

McFarlane, W.R., Stastny, P., & Deakins, S. (1992).Family-aided assertive community treatment: Acomprehensive rehabilitation and intensive casemanagement approach for persons with schizo-phrenic disorders. New Directions for MentalHealth Services, 53, 43-54.

McGrew, J.H., & Bond, G.R. (1995). Critical ingredi-ents of assertive community treatment: Judgmentsof the experts. Journal of Mental Health Adminis-tration, 22, 113-125.

McGrew, J.H., Bond, G.R., Dietzen, L.L. (1994).Measuring the fidelity of implementation of amental health program model. Journal of Consult-ing and Clinical Psychology, 62, 670-678.

Meisler, N. (1997, May). Practice guidelines for as-sertive community treatment. Paper presented atthe National Conference on Mental Health Statis-tics, Washington, DC.

Moncher, F.J., & Prinz, R.J. (1991). Treatment fi-delity in outcome studies. Clinical Psychology Re-view, 11, 247-266.

Morse, G.A., Calsyn, R.J., Allen, G. (1992). Experi-mental comparison of the effects of three treatmentprograms for homeless mentally ill people. Hospi-tal and Community Psychiatry, 43, 1005-1010.

Norusis, M.J. (1994). SPSS Professional Statistics6.1. Chicago: SPSS.

Nunnally, J.C. (1978) Psychometric theory (2nd ed.).New York: McGraw-Hill.

Olfson, M. (1990). Assertive community treatment:An evaluation of the experimental evidence. Hos-pital and Community Psychiatry, 41, 634-641.

Pedhazur, E.J. & Schmelkin, L.P. (1991). Measure-ment, design, and analysis: An integrated ap-proach. Hillsdale, NJ: Erlbaum.

Prirnm, A.B. (1996). Assertive community treatment.In William R. Breakey (Ed.), Integrated mentalhealth services: Modern community psychiatry(pp. 222-237). New York: Oxford UniversityPress.

Randolph, F.L. (1995, Spring). Improving systemsthrough systems integration: The ACCESS pro-gram. American Rehabilitation, 36-37.

Rosenheck, R.A., Morrissey, J., Lam, J., Calloway,M., Johnsen, M., Goldman, H., Randolph, F.,Blasinsky, M., Fontana, A., Calsyn, R., & Teague,G. (in press). Service system integration, access toservices, and housing outcomes in a program for

homeless persons with severe mental illness.American Journal of Public Health.

Rosenheck, R.A. & Neale, M.S. (in press). Cost-effectiveness of intensive psychiatric communitycare for high users of inpatient services. Archivesof General Psychiatry.

Rosenheck, R.A., Neale, M., Leaf, P., Milstein, R., &Frisman, L. (1995). Multi-site experimental coststudy of intensive psychiatric community care.Schizophrenia Bulletin, 21, 129-140.

Santos, A.B. (1996). Assertive community treatment.In Steve Soreff (Ed.), The seriously and persis-tently mentally ill: The state of the art treatmenthandbook (pp. 411-431). Seattle: Hogrefe, andHuber.

Santos, A.B., Deci, P.A., Lachance, K.R., (1993).Providing assertive community treatment forseverely mentally ill patients in a rural area. Hos-pital and Community Psychiatry, 44, 34—39.

Scott, A.G., & Sechrest, L. (1989). Strength of theoryand theory of strength. Evaluation and ProgramPlanning, 12, 329-336.

Stein., L.I., & Santos, A.B. (1998)Assertive commu-nity treatment of persons with serious mental ill-ness. New York: Norton.

Stein, L.I., & Test, M.A. (1980). Alternative to men-tal hospital treatment: I. Conceptual model, treat-ment program, and clinical evaluation. Archives ofGeneral Psychiatry, 37,392-397.

Teague, G.B., Drake, R.E., & Ackerson, T. (1995).Evaluating use of continuous treatment teams forpersons with mental illness and substance abuse.Psychiatric Services, 46, 689-695.

Test, M.A. (1992) Training in community living. InR.P. Liberman (Ed.), Handbook of psychiatric re-habilitation (153-170). Boston: Allyn and Bacon.

Test, M.A., Bond, G.R., McGrew, J.H., & Teague,G.B. (1997). P/ACT services research and fidelityto the model. Community Support Network News,11, 5-7.

Test, M.A., & Stein, L.I. (1976). Practical guidelinesfor the community treatment of markedly impairedpatients. Community Mental Health Journal, 12,72-82.

Thompson, S.T., Griffith, E.E.H., & Leaf, P.J.(1990). A historical review of the Madison modelof community care. Psychiatric Services, 41,625-634.

For reprints- Gregory B. Teague, PhD, Associate Professor and Chair, Department of Community Mental Health, Louis de laParte Florida Mental Health Institute, University of South Florida, 13301 Bruce B. Downs Boulevard, Tampa FL 33612 [e-mail:teague@finhi .usf.edu]