1.Journal of Brief TherapyVolume 5 Number 1 2006 Using Formal Client Feedback toImprove Retention and Outcome: Making Ongoing, Real-time Assessment FeasibleScott D. Miller Barry L. DuncanJeb Brown Ryan Sorrell Mary Beth Chalk Institute for the Study of Therapeutic Change Chicago, IllinoisResearch has found that client change occurs earlier rather than later in the treatment process, andthat the clients subject experience of meaningful change in the first few sessions is critical. Ifimprovement in the clients subject sense of well-being does not occur in the first few sessions thenthe likelihood of a positive outcome significantly decreases. Recent studies have found that thereare significant improvements in both retention and outcome when therapists receive formal, real-time feedback from clients regarding the process and outcome of therapy. However, the most usedinstruments in these feedback studies are long and take up valuable therapy time to complete. It hasbeen found that most therapists are not likely to use any feedback instruments if it takes more thanfive minutes to complete, score and interpret. This article reports the results of an evaluation of theuse of two very brief instruments for monitoring the process and outcome of therapy, the OutcomeRating Scale (ORS) and the Session Rating Scale (SRS), in a study involving 75 therapists and6,424 clients over a two year period. These two instruments were found to be valid and reliable andhad a high use-rate among the therapists. The findings are discussed in light of the current emphasison evidence-based practice.The proof of the pudding is in the eating. Cervantes, Don Quixote ***Outcome research indicates that the general trajectory of change in successfulpsychotherapy is highly predictable, with most change occurring earlier ratherthan later in the treatment process (Brown, Dreis, & Nace, 1999; Hansen &Lambert 2003). In their now classic article on the dose-effect relationship, Howard, Kopte,

2. Improving Retention and OutcomeKrause, and Orlinsky (1986) found that between 60-65% of people experienced significantsymptomatic relief within one to seven visitsfigures that increased to 70-75% after sixmonths, and 85% at one year. These same findings further showed, a course of diminishingreturns with more and more effort required to achieve just noticeable differences in patientimprovement as time in treatment lengthened (p. 361, Howard et al., 1986).Soon after Howard et al.s (1986) pioneering study, researchers began using earlyimprovementspecifically, the clients subjective experience of meaningful change in thefirst few visitsto predict whether a given pairing of client and therapist or treatmentsystem would result in a successful outcome (Haas, Hill, Lambert,Morrell, 2002;Lambert, Whipple, Smart, Vermeersch, Nielsen,Hawkins, 2001; Lueger, 1998; Lueger,2001). Continuing where they had left off, Howard, Lueger, Maling,Martinovich (1993)not only confirmed that most change takes place earlier than later, but also found that anabsence of early improvement in the clients subjective sense of well-being significantlydecreased the chances of achieving symptomatic relief and healthier life functioning bythe end of treatment. Similarly, in a study of more than 2000 therapists and thousands ofclients, Brown, et al. (1999) found that therapeutic relationships in which no improvementoccurred by the third visit did not on average result in improvement over the entire courseof treatment; this study further found that clients who got worse by the third visit weretwice as likely to drop out of treatment than clients who reported making progress. Moretelling, variables such as diagnosis, severity, family support, and type of therapy were, not. . . as important [in predicting eventual outcome] as knowing whether or not the treatmentbeing provided [was] actually working (p. 404).By the mid-nineties, researchers were using data generated during treatment to improvethe quality and outcome of care. In 1996, Howard, Moras, Brill, Martinovich, and Lutzshowed how measures of client progress could be used to determine the appropriateness ofthe current treatmentthe need for further treatment[and] prompt a clinical consultationfor patients who [were] not progressing at expected rates (p. 1063). That same year,Lambert and Brown (1996) made a similar argument using a shorter, and hence morefeasible, outcome tool.Other researchers had already found that clients early ratings of the alliance, likeprogress, were significant predictors of final treatment outcome (BachelorHorvath,1999, p. 139). Building on this knowledge, Johnson and Shaha (1996, 1997; Johnson,1995) were among the first to document the impact of outcome and process tools on thequality and outcome of psychotherapy as well as demonstrate how such data could foster acooperative, accountable relationship with payers.Several recent studies have documented significant improvements in both retention inand outcome from treatment when therapists have access to formal, real-time feedback fromclients regarding the process and outcome of therapy (DuncanMiller, 2000; Duncan,Miller,Sparks, 2004). For example, Whipple, Lambert, Vermeersch, Smart, Nielsen,and Hawkins (2003), found that clients whose therapists had access to progress and allianceinformation were less likely to deteriorate, more likely to stay longer, and twice as likely toachieve a clinically significant change. Formal client feedback has also been shown to beparticularly helpful in cases at risk for a negative or null outcome. A meta-analysis of threestudies by Lambert, Whipple, Hawkins, Vermeersch, Nielsen, and Smart (2003) found thatcases informed by client ratings of progress were, at the conclusion of treatment, better offthan 65% of those without access to such data (Average ES = .39). 3. Miller, Duncan, Brown, SorrellChalkThe present study was designed to assess the impact of two simple and brief, client-completed, rating scales of alliance and outcome on retention in and outcome from therapy.Research and clinical experience indicate that the length and complexity of the measuresemployed in the studies to date hinder their application in real world clinical settings (Miller,Duncan, Brown, Sparks,Claud, 2003). Indeed, Brown et al. (1999) found that themajority of practitioners are unlikely to use any measure or combinations of measures thattook more than five minutes to complete, score, and interpret. Therapists, it is clear, not onlyrequire valid and reliable but also feasible tools for inviting client feedback. As Lambert,Hansen, and Finch (2001) pointed out in a special issue of the Journal of Consulting andClinical Psychology on client feedback, treatment systems cannot tolerate expensive andtime-intensive markers of change, especially when used as a start up procedure or wherepatient (sic) progress is reported to therapists on a weekly schedule (p. 160).MethodParticipants The participants in the study were clients of Resources for Living (RFL), aninternational Employee Assistance Program (EAP) based in Austin, Texas. The companyemploys 75 in-house therapists who provide telephonic-based employee assistance,information and referral, executive coaching, individual therapy, disease management, andcritical incident services to 28 different corporate and organization customers. Therapistsat RFL range in age from 25 to 57, with an average age of 37.4, and are predominantlyfemale (72.2%). Average length of employment at RFL for those included in the studywas 3 years, with an average of 7 years of total clinical experience. The staff comes froma variety of professional disciplines, including clinical psychology (45%), social work(35%), and marriage and family therapy (20%), and the majority of them (92%) werelicensed to practice independently by their respective discipline. The clientele of RFL is culturally and economically diverse, including people ofAmerican, European, African, Latin, and Caribbean decent. In any given year, the severityof problems presented by clients of organization is comparable to those seen in a typicalmental health clinic, including anxiety, depression, alcohol and drug abuse, work andfamily issues, as well as chronic mental and physical health problems (Miller, Duncan,Brown et al., 2003). The sample in the present study included 6,424 clients that received telephonic basedcounseling between April 1, 2002 and March 31, 2004. In order to be included in thesample, the client must have received at least two sessions and completed an outcomequestionnaire by the end of each. Because callers have a right to remain anonymous,limited demographic information is available. Similar to most community mental healthoutpatient settings, two-thirds of the participants were female, one third male. The averageage of the sample was 36, with a median age of 34, mode of 20, and standard deviation of13. The level of distress as assessed by the outcome measure administered at intake wasalso similar to that found in a typical community mental health outpatient samplein fact,it was slightly greater than the figure reported by Miller, Duncan, Brown et al. 2003 (18.6versus 19.6). Given that the services offered by RFL are employer funded, it can be safely assumedthat all of the clients in the current study were either employed or were a family member 4. Improving Retention and Outcomeof someone who was working for a covered organization. The majority of clients whoutilized the service during the study period fell at the lowest end of the pay scale in theirrespective work settings, with 68% of the sample made up of line workers, 12% frommiddle and upper management, and 4% who had either retired or been terminated. Familymembers of a covered employee made up the remaining 16% of contacts. During the studyperiod, the top five presenting problems were: (1) marital (24.7%); (2) depression (10%);(3) anxiety (5.9%); (4) issues related to grief and loss (4.8%); and (5) drug and alcoholproblems (3.5%).Measures Client progress was assessed via the oral version of the Outcome Rating Scale (ORS[MillerDuncan, 2000]), a four-item, self-report instrument (see Appendix 1). The ORSwas developed as a brief alternative to the Outcome Questionnaire 45 (OQ-45)a popularbut longer measure developed by Lambert and colleagues (Lambert, Hansen, Umphress,Lunnen, Okiishi, Burlingame, HuefnerReisinger, 1996). Both scales are designed toassess change in three areas of client functioning widely considered valid indicators ofprogress in treatment: individual (or symptomatic) functioning, interpersonal relationships,and social role performance (work adjustment, quality of life [LambertHill, 1994]). In a recent issue of the Journal of Brief Therapy, Miller, Duncan, Brown et al. (2003)reported results of an initial investigation of the reliability and validity of the ORS. Pearsonproduct moment correlation between the ORS and the OQ-45 yielded a concurrent validitycoefficient of .58, a figure considered adequate given the brevity of the ORS. Reliabilityof the measure, as assessed by Cronbachs coefficient alpha, was .93, test-retest reliabilityat the second session, .66. Independent confirmation of the reliability of the ORS wasconducted by the Center for Clinical Informatics11 using data collected at RFL. In thissample, coefficient alpha was found to be .79 (n = 15,778), while test-retest reliability atsecond administration was .53 (n = 1,710). With regard to the latter, it is important to notethat lower test-retest reliability is expected for measures designed to be sensitive to changefrom week to week as research has shown both the ORS and OQ-45 to be (Miller, Duncan,Brown et al. 2003; Vermeersch, Lambert,Burlingame, 2000). The therapeutic alliance was assessed via the oral version of the Session Rating Scale3.0 (SRS [Miller, Duncan,Johnson, 2000] see Appendix 2). The SRS is a brief, four-item,client-completed measure derived from a ten-item scale originally developed by Johnson(1995). Items on this measure reflect the classical definition of the alliance first stated byBordin (1979), and a related construct known as the clients theory of change (Duncan Miller, 2000). As such, the scale assesses four interacting elements, including the qualityof the relational bond, as well as the degree of agreement between the client and therapiston the goals, methods, and overall approach of therapy. To test the reliability and validity of the SRS, Duncan, Miller, Reynolds, Sparks,Claud, Brown,Johnson (2004) compared the instrument to the Revised HelpingAlliance Questionnaire (HAQ-II), a widely used measure of therapeutic alliance. Thereliability for the SRS compared favorably with the HAQ-II (.88 and .90, respectively).Test-retest reliability for the SRS over six administrations was .74, compared to .69 forthe HAQ-II. Concurrent validity as estimated by Pearson product moment correlationsaveraged .48, evidence that the SRS and HAQ-II are referencing similar domains. As with 5. Miller, Duncan, Brown, SorrellChalkthe ORS, independent confirmation of the reliability of the SRS was conducted by theCenter for Clinical Informatics using data collected at RFL. In a sample of nearly 15,000administrations, coefficient alpha was found to be .96, remarkably high for a four-itemmeasure. Test-retest reliability was .50, comparable to that of the ORS.Procedures The study was divided into four distinct phases: (1) initial training (including severalsite visits by the first two authors over a 6-month period); (2) baseline data collection andanalysis (6 months); (3) implementation of automated feedback condition (6 months); and(4) continued evaluation (12 months). During the first phase, therapists were trained onsite by the first two authors in the proper administration of the ORS and SRS. Both toolswere then incorporated into RFLs existing, computerized client tracking system, makingthe use of the scales a uniform and automatic process along with routine record keeping. Inpractice, the ORS was completed at the start of each session and the SRS at the end. During the second phase, baseline data from the ORS and SRS was collected for1,244 clients that received two or more telephonic counseling sessions. Gathering suchdata was a critical step in developing the clinical norms that would form the basis forthe automated feedback system known as SIGNAL (Statistical Indicators of Growth,Navigation, Alignment and Learning). As the name implies, the WindowsTM-based systemused a traffic light graphic to provide real-time warnings to therapists when an individualclients ratings of either the alliance or outcome fell significantly outside of the establishednorms. As an example of the kind of feedback a therapist would receive when a particularclients outcomes fell outside of the expected norms, consider Figure 2. The dotted linerepresents the expected trajectory of change for clients at RFL whose total score at intakeon the ORS is 10. Consistent with prior research and methodology (LambertBrown,2002), trajectories of change were derived via linear regression, and provide a visualrepresentation of the relationship between ORS scores at intake and at each subsequentadministration. Colored bands corresponding to the 25th (yellow) and 10th (red) percentilesmark the distribution of actual scores below the expected trajectory over time. The horizontaldashed-dotted line at 25 represents the clinical cutoff score for the ORS. Scores fallingabove the line are characteristic of individuals not seeking treatment and scores belowsimilar to people who are in treatment and likely to improve (Duncan, Miller, Reynolds,et al. 2003). The remaining solid line designated the clients actual score from session tosession. As can be seen in Figure 2, the clients score at the second session falls below the25th percentile. By session 3 the score has fallen even further, landing in the red arearepresenting the 10th percentile in the distribution of actual scores. As a result, the therapistreceives a red signal, warning of the potential for premature drop out and an increasedrisk for a negative or null outcome should therapy continue unchanged. An option buttonprovides suggestions for addressing the problem, including: (1) talking with the client aboutproblems in the alliance; (2) changing the type and amount of treatment being offered; and(3) recommending consultation or supervision. 6. 10Improving Retention and OutcomeFigure 2: SIGNAL Outcome Feedback Client feedback regarding the alliance was presented in a similar fashion at the endof each visit (see Figure 3). A solid line designates a clients actual score from session tosession. Colored bands represent the 25th (yellow) and 10th (red) percentile of respondersin the study sample (Duncan, Miller, Reynolds et al. 2004). In this particular example, theclient scores a 34 on the SRS at the conclusion of the first visit. As can be seen, this scorefalls below the 25th percentile thus triggering a yellow signal. Given the relative rarity ofsuch a score, the therapist is advised to check in with the client about their experience,express concern for their work together, and explore options for changing the interactionbefore ending the session. Once the normative data was collected and the SIGNAL feedback system created,the study entered its third phase. Outcome and alliance scores were entered and SIGNALfeedback given for the next 1568 clients that sought services at RFL. During this time, ahandful of site visits by the first two authors, plus ongoing support from administration andmanagement at RFL encouraged a high rate of compliance with and consistency in the useof the measures and SIGNAL system. In the fourth and final phase of the study, data wascollected from an additional 3,612 clients, providing a large sample by which the effect offeedback on the retention in and outcome from clinical services could be evaluated.Data AnalysisThe outcome of treatment was assessed in three ways. First, a continuous variable 7. Miller, Duncan, Brown, SorrellChalk 11Figure 3: SIGNAL Alliance Feedbackgain score was calculated by subtracting the ORS score at intake from ORS score at thefinal session. Second, a residualized gain score was computed based on a linear regressionmodel. Residualized gain scores are necessary whenever intake scores are correlated withchange scores in order to control for the change in gain scores associated with differencesin ORS scores at intake (CohenCohen, 1983; CampbellKenny, 1999). Third, andfinally, a categorical variable classification of outcome as improved, unchanged, ordeteriorated was determined by comparing the gain score against the reliable changeindex of the ORS (RCI = 5; DuncanMiller, 2004). The RCI for the ORS is 5, so caseswith a gain score greater than +5 were classified as improved, -5 as worse, and those fallingbetween + or 4 points as unchanged. In order to facilitate interpretation of the magnitude of improvement across phases,gain scores were converted to effect sizes (SmithGlass, 1987). The effect sizes in thepresent study were calculated by dividing the gain score by the standard deviation of theORS in a non-treatment normative sample (Miller, Duncan, Brown et al. 2003). As such,the effect sizes reported can be interpreted as an indication of how much clients in the studyimproved relative to a normal population. Finally, the relationship between the alliance and outcome was also subjected to analysis.Given prior research showing that clients early ratings of the alliance are significantpredictors of final treatment outcome, simple correlations were computed between SRSscores at intake and end of treatment gain scores. To determine the effect of improvingalliances on the outcome of treatment, gain scores for the SRS were computed and thencorrelated with gain scores on ORS. Finally, the relationship between the actual use of the 8. 12 Improving Retention and OutcomeSRS and outcome from and retention in treatment the first session was also examined.Results Data gathered during the baseline phase of the study revealed that the majority ofclients (56%) who received two or more sessions did not remain with the same therapistover the course of services. Analysis of the outcome data from this period further showedthat clients who switched therapists fared significantly worse than those treated by thesame therapist from session to session (Effect Size = .02 versus .79). Reflecting on thepossible causes of the rampant switching, therapists and administrators identified officialagency policy favoring immediate access over continuity of services. Prior to enteringthe third phase of the study during which SIGNAL was launched, the policy was changed.Thereafter, therapists were strongly encouraged to retain clients by setting aside a certainamount of time per week during which standing appointments could be scheduled. By the last phase in the study, the number of clients that switched therapists had beencut in half (~27%). The outcomes for clients who stayed with the same therapist comparedto those who switched can be found in Table 1. Across phases, clients who stayed with thesame therapist from session to session fared significantly better. Note, additionally, thatthe overall outcomes of both groups improved over time. Progress was most pronouncedin the group of clients that switched therapists, going from an effect size of .02 at baselineto .40 by the end of the final evaluation phase (p.001). Clients who remained with thesame therapist also improved significantly over the course of the study, with an initialeffect size of .79 at baseline increasing to .93 during the last phase of the study (p.01). % of MeanSwitchedSampleMean ORS Mean Effect Time period sample in Residualtherapist size at intake gain scoreSizetime period gain scoreBaseline695 56% 18.30.13-4.60.02Intervention689 44% 18.3 1.9-2.90.28Evaluation993 27% 18.3 2.7-2* 0.40Sametherapist Baseline 549 44% 18.35.40.97 0.79 Intervention 879 56% 18.95.9 1.5 0.87 Evaluation2719 73% 19.26.320.93 * p.01 when comparing result to baseline period using two tailed t-test of significanceTable 1: Comparisons between clients that switched therapist or stayed with therapistTable 2 presents the mean ORS intake scores, gain scores, and residualized gain scoresacross the various phases of the study. As can be seen, the magnitude of improvement issubstantial, with the overall effect size of treatment more than doubling from the baselineperiod to the final evaluation phase (baseline ES = .37 versus final phase ES = .79). Aone-way analysis of variance further found the difference between residualized gain scoresacross phases was highly significant (p.001). 9. Miller, Duncan, Brown, SorrellChalk 13MeanMean ORS Mean gainTime period Sample size at intake score ResidualEffect Sizegain scoreBaseline phase:(six months) 1244 18.32.5 -2.3 0.37Interventional phase:1-6 months post SIGNAL System1568 18.64.2-0.5* 0.62Evaluation phase:7-18 months post SIGNAL System 361219 5.41*0.79* p.001 when comparing result to baseline period using two tailed T-test of significance Table 2: Outcomes for all cases across phasesTable 3 presents the results from the categorical variable classification of outcome.Using the RCI of the ORS as the criteria by which change was assessed, the data indicatethat improvement in outcomes was due to a 13% increase in the percentage of clientsreporting significant improvement and an 11% decrease in cases reporting deterioration.With regard to the latter, it is important to note that the relatively high percentage of clientsthat deteriorated during the various phases resulted from the inclusion of clients who scoredin the normal range on the ORS (Total Score25) at their initial visit. Indeed, 25% of theRFL sample had intake scores at or above the established clinical cutoff for the measure.In general, clients scoring in the normal range on standardized outcome measures at intaketend to average little or no improvement or even worsen with treatment (Brown et al.1999; Brown, Burlingame, Lambert, Jones,Vacarro, 2001; Duncan, Miller,Sparks,2004). When the analysis of the data in the present study is limited to clients falling inthe clinical rangeas is usually the case in controlled clinical trials of psychotherapy ormedicationsthe rate of deterioration not only decreases in the final phase to 5% but theoverall effect size for the same time period increases considerably. In the final phase, forexample, the overall effect size is .79. However, with the sample restricted to clinical rangecases only, the effect size increases to 1.06. Time period% improved % unchanged % worseBaseline phase:34%47% 19%Interventional phase 42%46% 12%Evaluation phase 47%45%8%Table 3: Categorical evaluation of outcomes across phases Turning to the analysis of the relationship between alliance and outcome, scores onthe SRS at intake proved to be a weak predictor of change on the ORS (p.05). However,increases in SRS scores over the course of treatment were associated with better outcomes.For all cases, gain scores on the SRS correlated .13 with gain scores on the ORS (n=4785,p.0001). As was also expected, the relationship was even stronger for clients whose SRSscores at intake fell below the clinical cutoff of 36 established for the SRS (Duncan, Miller,Reynolds, et al. 2004). Since correlation does not imply causality, the relationship between SRS and ORSscores was examined to determine whether there was any impact of distress on clientsratings of the alliance. The analyses showed that client ratings on the ORS and SRSwere correlated both at the beginning and end of treatment (r = .10 and .19, respectively).Thus, clients who were less distressed were more likely to rate the alliance higher. From 10. 14 Improving Retention and Outcomethe present data, it is not possible to determine whether feeling better leads to betteralliances or better alliances result in feeling better. It seems plausible to assume that thereis a reciprocal effect, with improved alliances leading to decreases in distress leading toimproved alliance and so on. What can be said with greater certainty is that the simple act of monitoring the alliancehas a beneficial impact on outcomes and retention rates. During the baseline period, forexample, 20% of the cases with ORS scores at intake did not have SRS scores for thatvisit. Such cases were three times less likely to have an additional session than those forwhich alliance data was present (6% versus 19%, respectively). Failure to complete theSRS was also associated with less change on the ORS at the end of treatment. Amongclients who remained with the same therapists throughout treatment, those that completedthe SRS at intake averaged 3.3 points more change (residualized gain score) than thosethat did not (p.01; two tailed t-test). By the final evaluation period, utilization rates forthe SRS by therapists at RFL had improved so much that failure to complete the measurewas no longer predictive of drop out after the first session. Even in this phase, however,failure to complete the SRS was associated with less change by the end of treatment (Meanresidualized change score = 1, p.05; two tailed t-test).Discussion The present study found that providing formal, ongoing feedback to therapistsregarding clients experience of the alliance and progress in treatment resulted in significantimprovements in both client retention and outcome. To summarize briefly, access to theclients experience of progress in treatment effectively doubled the overall effect sizeof services (See Figure 1). And while high alliance scores were only weakly related tooutcome, improving a poor alliance during at the outset of treatment was correlated withsignificantly better outcomes at the conclusion. At the same time, clients of therapists whofailed to seek feedback regarding the alliance as assessed by the SRS were three times lesslikely to return for a second session and had significantly poorer outcomes. Figure 1: Effect Size 11. Miller, Duncan, Brown, SorrellChalk 15 As dramatic as these results may seem, they are, as noted in the Introduction sectionof this article, entirely consistent with findings from similar research (Lambert et al. 2003;Whipple et al. 2003). One important difference between the current study and previousresearch is the simplicity and feasibility of the measures employed. Here, the results arealso compelling. For example, of the thousands of sessions of treatment provided duringthe course of this study, only a handful of complaints were logged from clients regardingthe scales. At the same time, use of the scales by therapists to inform treatment was, bythe final phase of the study, exceptionally high as compared to studies that have employedlonger, more complicated measures (~99% versus ~25% at 1 year [Miller, Duncan, Brownet al. 2003]). The findings from the present study are limited by a number of factors. First andforemost is the reliance on client self-report measures (BouletBoss, 1991). Clearly,evaluation of outcome and alliance via the ORS and SRS is far from comprehensive anddoes not contain multiple perspectives (e.g., therapists, outside judges, objective criteria,etc.). At the same time, however, both measures are similar in scope to those being usedin patient-focused as opposed to traditional efficacy types of research studies (Lambert,2001). A second issue to consider when determining the generalizability of the results isthe type of treatment services examined in the present study. Although the sample didnot differ either demographically or in terms of measured levels of distress at intake, allservices were offered via the telephone. Provision of clinical services via the telephoneand other technologies (e.g., internet, video-conferencing) has increased dramaticallyover the last two decades. Although fewer studies have been done overall, research todate finds such services work for the same reasons as (BobevskiMcLennan, 1998) andproduce results roughly equivalent to face-to-face treatment for a number of presentingconditions, including the promotion of health related behaviors, anxiety and depression,obsessive-compulsive disorder, medication and case management, and suicide prevention(Gold, AndersonSerxner, 2000; King, Nurcombe, Bickman, Hides, Reid, 2003; Ko Lim, 1996; Liechtenstein, Glasgow, Lando, Ossip-Klein, et al. 1996; Reese, Conoley Brossart, 2002; Salzer, Tunner, Charney, 2004; Taylor, Thordarson, Spring, Yeh, Corcoran,Eugster,Tisshaw, 2003; X DaySchneider 2003). Still, more studies involving directcomparisons are needed in order to better establish equivalence. On a positive note, researchon feedback derived from the ORS and SRS is currently underway in a number of settingsthat provide face-to-face services, including residential and intensive-outpatient substanceabuse treatment, outpatient community mental health, and a college counseling center. One final issue that merits discussion is the drop out rate of clients served by RFL.Although the average number of sessions for clients who returned for at least two visitswas similar to national retention rates (Mean = 3.5), the percentage of clients having onlyone treatment contact was significantly greater (~ 80% versus 30% [Talmon, 1990]). Inorder to understand the reasons for and any impact of this difference on client satisfactionand outcome, a survey was conducted (Sorrell, MillerChalk, in press). Follow up phonecalls were made to a random sample totaling twenty percent of clients who had contactedRFL for services a single time during July 2002 (n = 225). The average length of timebetween contact and follow up was 58 days, with a range from 37 to 77. During the interview, each former client was asked to complete the ORS and answertwo opened ended questions regarding the services received. Interestingly, scores on the 12. 16Improving Retention and Outcomeoutcome measures showed that that the large majority of those attending a single session(80%) had made positive change (average effect size = 1.3) while the remaining 20%experienced either no change or had deteriorated. Table 4 summarizes the responses clientsgave when they were asked why they had not sought out further care with RFL. Consistentwith the results on the ORS, clients who rated improved were much more likely than thosereporting a negative or null outcome to cite lack of need as the major reason for notseeking further services. On other hand, clients reporting a negative or null outcome werefour times more likely than those rating improved to cite counselor unhelpfulnessaresult also consistent with results on the ORS. No reason Too busyWanted toGetting Counselor SystemOutcome to call to callhandle ithelp didnt help failure backback myself elsewherePositive 44% 20%6% 11% 8%11%Negative 24% 27%23%10% 8%8% or null Table 4: Single session follow up study When clients rating improved were asked to account for the changes they hadexperienced since their single session, 28% cited talking to the RFL counselor as themajor contributorsecond only to extratherapeutic factors (36%). Only 10% of such clientsattributed the change to another treatment service or provider. Indeed, whether clients ratedimproved or had experienced a negative or null outcome, very few found it necessaryor desirable to seek out treatment elsewhere (8% for both groups). Such findings, whenconsidered together with the results from the ORS cited above, indicate that, whateverthe cause, the high percentage of single session contacts in the present study cannot beattributed to poor outcome or quality of service. Turning briefly to the implications of the present findings for the practice of therapy,the field has long sought to establish itself on solid ground through the creation of a reliablepsychological formularyprescriptive treatments for specified conditions. Such effortshave only intensified in recent years given the harsh economic climate of the Americanhealthcare system (DuncanMiller, 2000). Thus, phrases such as evidence-basedpractice, empirically supported treatments, and the like have come to characterize the bestthat clinical practice has to offer. The assumption inherent in current efforts is that a unified or systematic application ofscientific knowledge will lead to a universally accepted standard of care that, in turn, resultsin more effective and efficient treatment. Few would debate the success of this perspective inmedicine where an organized knowledge base, coupled with improvements in diagnosis andpathology, and the development of treatments containing specific therapeutic ingredients,have led to the near extinction of a number of once fatal diseases. Unfortunately, for all theclaims and counterclaims, and thousands of research studies, the field of therapy, in spite ofa numerous years of research and development, can boast of no similar accomplishments.Indeed, available research evidences calls the validity of this entire way of thinking about,organizing, and operationalizing clinical practice into question. To briefly summarize the data, virtually hundreds of studies conducted by different 13. Miller, Duncan, Brown, SorrellChalk 17researchers, using a variety of measures, and increasingly sophisticated research designsprovide little, no, or contradictory evidence that: Models of therapy contain specific therapeutic ingredients or exert diagnosticspecific benefits (AsayLambert, 1999; Wampold, 2001); Models of treatment differ in terms of outcome (Hubble, Duncan,Miller, 1999;Wampold, 2001); Treatment manuals, when strictly adhered to by practitioners, improve the qualityor outcome of therapy (Adis, Wade,Hatgis, 1999; Beutler, Malik, Alimohammed,Harwood, Talebi, Noble,Wong, 2004; LambertOgles, 2004; Shadish, Matt, Navarro, Phillips (2000); Quality assurance practices either improve the quality or the outcome of treatment(JohnsonShaha, 1996, 1997); or that Training in psychotherapy reliably improves success (AtkinsChristensen, 2001;LambertBergin, 1994; LambertOgles, 2004; Weisz, Weiss, Alicke,Klotz, 1987). Despite these research findings, many therapy practitioners and researchers stillfind it attractive to attempt to fit the round peg of psychotherapy into the square hole ofmedicine. Indeed, the general acceptance of the medico-scientific perspective in Westernsociety makes it easy to see how anything short of emulating the fields seemingly morescientifically minded and financially successful cousins in medicine is viewed as courtingmarginalization. As Nathan (1997) argued in the Register Report, therapists need to put[their] differences aside, find common cause, and join together to confront a greater threat.securing the place of psychological therapy in future health care policy and planning (p.5). Still, the facts are difficult to ignore: psychotherapy does not work in the same way asmedicine. The improvements in outcome hoped for and promised by the identification,organization, and systematization of therapeutic process have not materialized. In truth, however, consumers (and payers) care little about how change comes aboutthey simply want it and in the most accessible format possible. As such, the fields exclusivefocus on the means of producing change (i.e., models, techniques, therapeutic process)has been and continues to be on the wrong track. Consider the results of focus groupsconducted by the American Psychological Association (APA, 1998). When asked, 76% ofpotential consumers of psychotherapy identified low confidence in the outcome of therapyas the major reason for not seeking treatment, far eclipsing variables traditionally thoughtto deter people from seeing a therapist (e.g., stigma, 53%; length of treatment, 59%; lackof knowledge, 47%). Worse yet, a recent survey of 3,500 randomly selected subscribersof Readers Digest rated psychologists and social workers below auto mechanics and taxidrivers and only slightly above lawyers in trustworthiness (Psychotherapy in Australia,2000). The present study adds to a growing literature on a different approach to effective,efficient, and accountable treatment practice. Instead of assuming that identifying andutilizing the right process leads to favorable results, these efforts use outcomespecifically,client feedbackto both inform and construct treatment as well as inspire innovation. Putanother way, rather than evidence-based practice, therapists tailor services to the individualclient via practice-based evidence; instead of empirically supported therapies, consumerswould have access to empirically validated therapists. Whether the field can put outcome 14. 18 Improving Retention and Outcomeahead of process, given its historical and current emphasis on identifying and codifying themethods of treatment remains to be seen. As Lambert et al. (2004) points out, however,those advocating the use of empirically supported psychotherapies do so on the basis ofmuch smaller treatment effects (p. 296).ReferencesBachelor, A.,Horvath, A. (1999). The therapeutic relationship. In M.A. Hubble, B.L. Duncan,S.D. Miller (Eds.). The heart and soul of change: What works in therapy. Washington, D.C.: American Psychological Association Press, 133-178.Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy, 16, 252-260.Brown, G.S., Burlingame, G.M., Lambert, M.J., Jones, E.Vacarro, J. (2001). Pushing the quality envelope: A new outcomes management system. Psychiatric Services, 52 (7), pp 925-934.Brown, J., Dreis, S.,Nace, D.K. (1999). What really makes a difference in psychotherapy outcome? Why does managed care want to know? In M.A. Hubble, B.L. Duncan, and S.D. Miller (Eds.). The heart and soul of change: What works in therapy (pp. 389-406). Washington, D.C.: American Psychological Association Press, Boulet, J.,Boss, M. (1991). Reliability and validity of the Brief Symptom Inventory. Journal of Consulting and Clinical Psychology, 3(3), 433-437.Campbell, DTKenny, DA (1999). A Primer on regression artifacts. New York: Gilford Press.Cohen, JCohen, P (1983). Applied multiple regression/correlational analysis for the behavioral sciences. Hillsdale, NJ.: Lawrence ErlbaumAssociates.Duncan, B.L.,Miller. S.D. (2000). The heroic client: Principles of client-directed, outcome-informed therapy. San Francisco: Jossey-Bass.Duncan, B.L., Miller, S.D., Reynolds, L., Sparks, J., Claud, D., Brown, J.,Johnson, L.D. (2004). The session rating scale: Psychometric properties of a working alliance scale. Journal of Brief Therapy, 3(1).Duncan, B.L., Miller. S.D.,Sparks, J. (2004). The heroic client: Principles of client- directed, outcome -informed therapy (revised). San Francisco: Jossey-Bass.Hansen, N.B.,Lambert, M.J. (2003). An evaluation of the dose-response relationship in naturalistic treatment settings using survival analysis. Mental Health Services Research, 5, 1-12.Haas, E., Hill, R. D., Lambert, M. J.,Morrell, B. (2002). Do early responders to psychotherapy maintain treatment gains? Journal of Clinical Psychology, 58, 1157- 1172.Howard, K.I., Lueger, R. J., Maling, M.S.,Martinovich, Z. (1993). A phase model of psychotherapy outcome: Causal mediation of change. Journal of ConsultingClinical Psychology, 61, 678-685.Howard, K.I, Moras, K., Brill, P.L., Martinovich, Z.,Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient Progress. American Psychologist, 51, 1059-1064.Howard, K.I., Kopte, S.M., Krause, M.S.,Orlinsky, D.E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 41, 159-164.Johnson, L.D. (1995). Psychotherapy in the age of accountability. New York: Norton. 15. Miller, Duncan, Brown, SorrellChalk 19Johnson, L.,Shaha, S. (1996). Improving quality in psychotherapy. Psychotherapy, 35,225-236.Johnson, L.D.,Shaha, S.H. (July, 1997). Upgrading clinicians reports to MCOs.Behavioral Health Management, 42-46.Lambert, M.J. (2001). Psychotherapy outcome and wuality improvement: Introductionto the special section on patient-focused research. Journal of Consulting and ClinicalPsychology, 69(2), 147-149.Lambert, M.J.,Brown, G. S. (1996). Data-based management for tracking outcome inprivate practice. Clinical Psychology: Science and Practice, 3, 172-178.Lambert, M.J., Hansen, N.B.,Finch, A.E. (2001). Patient-focused research: Usingpatient outcome data to enhance treatment effects. Journal of Consulting and ClinicalPsychology, 69(2), 159-172.Lambert, M. J., Hansen, N.B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M.,Huefner, J., Reisinger, C. (1996). Administration and scoring manual for the OQ 45.2. Stevenson, MD:American Professional Credentialing Services.Lambert, M.J., Whipple, J.L., Hawkins, E.J., Vermeersch, D.A., Nielsen, S.L.,Smart,D.W. (2003). Is it time for clinicians routinely to track patient outcome? A meta-analysis. Clinical Psychology, 10, 288-301.Lambert, M.J., Whipple, J.L., Smart, D.W., Vermeersch, D.A., Nielsen,S.L., Hawkins,E.J. (2001). The effects of providing therapists with feedback on patient progressduring psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-68.Lambert, M..J.,Hill C.E. (1994). Assessing psychotherapy outcomes and processes.In A.E. Bergin,S.L. Garfield (eds.). Handbook of psychotherapy and behaviorchange. New York: John Wiley, 72-113.Lueger, R. J. (1998). Using feedback on patient progress to predict the outcome ofpsychotherapy. Journal of Clinical Psychology, 54, 383-393.Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E.Grissom, G.(2001). Assessing treatment progress of individual patients using expected treatmentresponse models. Journal of Consulting and Clinical Psychology, 69, 150-158.Miller, S.D.,Duncan, B.L. (2000b). The outcome rating scale. Chicago, IL: Authors.Miller, S.D., Duncan, B.L., Brown, J., Sparks, J.,Claud, D. (2003). The outcome ratingscale: A preliminary study of the reliability, validity, and feasibility of a brief visualanalog measure. Journal of Brief Therapy, 2(2), 91-100.Miller, S.D., Duncan, B.L.,Johnson, L.D. (2000). The session rating scale 3.0. Chicago,IL: Authors.Smith, M.L.,Glass, G.V. (1987). Research and evaluation in education and the socialsciences. Englewood Cliffs, New Jersey: Prentice Hall.Talmon, M. (1990). Single session therapy. San Francisco, CA: Jossey-Bass.Vermeersch, D.A., Lambert, M.J.,Burlingame, G.M. (2000). Outcome questionnaire:Item sensitivity to change. Journal of Personality Assessment, 74, 242-261.Whipple, J.L., Lambert, M.J., Vermeersch, D.A., Smart, D.W., Nielsen, S.L.,Hawkins,E.J. (2003). Improving the effects of psychotherapy: The use of early identificationof treatment and problem-solving strategies in routine practice. Journal of CounselingPsychology, 50, 59-68.1Center for Clinical Informatics, 1821 East Meadowmoor Road, Salt Lake City, UT 84117. On the web at: www.clinical-informatics.com. 16. 20 Improving Retention and OutcomeScott D. MillerBarry L. DuncanJeb BrownRyan SorrellMary Beth ChalkInstitute for the Study of Therapeutic ChangeP.O. Box 578264Chicago, Illinois 60657-8264scottdmiller@talkingcure.com Appendix 1 Scripting for Oral Administration of Outcome Rating Scale*Im going to ask some questions about four different areas of your life, includingyour individual, interpersonal, and social functioning. Each of these questions is basedon a 1 to 10 scale, with 10 being high (or very good) and 1 being low (or very bad).Thinking back over the last week (or since our last conversation), how would yourate:1. How you have been doing personally? (On the scale from 1 to 10)a. If the client asks for clarification, you should say your self, you as anindividual, your personal functioning.b. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y?c. If the client gives one number for one area of personal functioning and offersanother number for another area of functioning, then ask the client for anaverage.2. How have things been going in your relationships? (On the scale from 1 to 10)a. If the client asks for clarification, you should say in your family, in your closepersonal relationships.b. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y?c. If the client gives one number for one family member or relationship type andoffers another number for another family member or relationship type, then askthe client for an average.3. How have things been going for you socially? (on the scale from 1 to 10)a. If the client asks for clarification, you should say, your life outside the home orin your community, work, school, church.b. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y?c. If the client gives one number for one aspect of his/her social functioning and 17. Miller, Duncan, Brown, SorrellChalk 21 then offers another number for another aspect, then ask the client for an average.4. So, given your answers on these specific areas of your life, how would you ratehow things are in your life overall? The clients responses to the specific outcome questions should be used to transitioninto counseling. For example, the counselor could identify the lowest score given andthen use that to inquire about that specific area of client functioning (e.g., if the clientrated the items a 7, 7, 2, 5, the counselor could say from our responses, it appearsthat youre having some problems in your relationships. Is that right?) After that, thecounseling proceeds as usual.*Copies can be obtained from the Institute for the Study of Therapeutic Change at: www.talkingcure.comAppendix 2Scripting for Oral Administration of Session Rating Scale* Im going to ask some questions about our session today, including how well youfelt understood, the degree to which we focused on what you wanted to talk about, andwhether our work together was a good fit. Each of these questions is based on a 1 to 10scale, with 10 being high (or very good) and 1 being low (or very bad).Thinking back over our conversation, how would you rate:1. On a scale of 1-10, to what degree did you feel heard and understood today, 10being completely and 1 being not at all?a. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y?b. If the client gives one number for heard and another for understood, then ask theclient for an average.2. On a scale of 1-10, to what degree did we work on the issues that you wantedto work on today, 10 being completely and 1 being not at all?a. If the client asks for clarification, you should ask, did we talk about what youwanted to talk about or address? How well on a scale from 1 10?b. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y?3. On a scale of 1-10, how well did my approach, the way I worked, make senseand fit for you?a. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y? 18. 22 Improving Retention and Outcomeb. If the client gives one number for make sense and then offers another number forfit, then ask the client for an average.4. So, given your answers on these specific areas, how would you rate how thingswere in todays session overall, with 10 meaning that the session was right foryou and 1 meaning that something important that was missing from the visit?a. If the client gives you two numbers, you should ask, which number would youlike me to put? or, is it closer to X or Y? *Copies can be obtained from the Institute for the Study of Therapeutic Change at: www.talkingcure.com