Psychotherapy Volume 31/Summer 1994/Number 2

APPLYING OUTCOME RESEARCH: INTENTIONALUTILIZATION OF THE CLIENT'S FRAME OF REFERENCE

BARRY L. DUNCAN DOROTHY W. MOYNIHANDayton Institute for Family Therapy

Centerville, Ohio

The percentage of outcome varianceattributable to extratherapeutic andcommon factors, and the superiority ofclient's predictions of outcome,challenges an emphasis on theoreticalframes of reference and offers acompelling argument for allowing theclient to direct the psychotherapeuticprocess. This article suggests thattherapists intentionally utilize the client'sframe of reference for the explicitpurpose of influencing successfuloutcome. A proposal for a client-directedprocess is offered that de-emphasizestheory and seeks deliberate enhancementof common factor effects and maximumcollaboration with the client through allphases of intervention.

A review of the outcome research (Lambert,Shapiro & Bergin, 1986) suggests that 30% ofoutcome variance is accounted for by the commonfactors (variables found in a variety of therapiesregardless of the therapist's theoretical orienta-tion). Techniques (factors unique to specific ther-apies) account for 15% of the variance, as doexpectancy/placebo effects (improvement re-sulting from the client's knowledge of being intreatment) (Lambert, 1992). Accounting for theremaining 40% of the variance are the extrathera-peutic change variables (factors that are part of

Correspondence regarding this article should be addressedto Barry L. Duncan, 747 Hidden River Drive, Port St. Lucie,FL 34983.

the client and his/her environment that aid in re-covery) (Lambert et al., 1986).

Lambert (1992) suggests that most of the suc-cess gleaned from intervention can be attributedto the common factors. Common factors havebeen conceptualized in a variety of ways. A recentanalysis of the common factors literature (Gren-cavage & Norcross, 1990) revealed that the mostfrequently addressed commonality was the devel-opment of a collaborative therapeutic relation-ship/alliance.

Supportive of the Lambert et al. (1986) reviewis Patterson's (1984) report that empathy, respect,and genuineness account for from 25% to 40%of outcome variance. Patterson (1989) concludesthat the outcome research undercuts the view thatexpertise in methods and techniques is the criticalfactor in promoting change; rather, the evidencesuggests that therapists' influence lies in provid-ing the conditions under which the client engagesin change (Patterson, 1989).

The outcome literature challenges the inherentand invariant validity of a specific orientation,given that specific technique seems largely insig-nificant when compared with common factors andextratherapeutic variables. Another empiricalchallenge to the therapist's frame of reference isprovided by research demonstrating that clientperceptions of therapist-provided variables are themost consistent predictor of improvement (Gur-man, 1977; Horowitz et al., 1984). More re-cently, the therapist-provided variables have beenstudied in terms of the therapeutic alliance, whichincludes both therapist and client contributions tothe therapeutic climate, and emphasizes collabo-ration between therapist and client in achievingthe goals of therapy (Marmar et al., 1986).

A recent study by Bachelor (1991) exploredthe contribution to improvement of three alliancemeasures and focused on the perceptions of theclient and therapist. Confirming and adding em-

294

Applying Outcome Research

phasis to many previous findings regarding com-mon factors and the client's perceptions, Bachelor(1991) found that client perceptions yield strongerpredictions of outcome than therapists, and thatfrom the client's view, the most salient factorsare therapist-provided help, warmth, caring,emotional involvement, and efforts to explore rel-evant material.

The significance of client (extratherapeutic)and common factors as well as the superiority ofclients' perceptions in predicting outcomes offera compelling argument for more attention to theclient's resources and experience of the psycho-therapeutic process (Rogers, 1957). As an attemptto further apply outcome research, this article pro-poses an intentional utilization of the client'sframe of reference for the explicit purpose of in-fluencing successful outcome. The client's frameof reference is discussed in terms of: 1) the cli-ent's perceptions and experience of the therapeu-tic relationship; and 2) the client's perceptionsand experience of the presenting complaint, itscauses, and how therapy may best address theclient's goals, i.e., the client's informal theory(Held, 1991). A client-directed process in psy-chotherapy is presented that de-emphasizes theo-retical frames of reference and seeks deliberateenhancement of common factor effects and maxi-mum collaboration with the client through allphases of intervention.

The Client's Frame of Reference:The Therapeutic Relationship

One way of enhancing common factor effectsis to extend the definitions of therapist-providedvariables to include the client's perception of thetherapist's behavior. Another way is to examinetherapist assumptions critically and eliminatethose that may undermine the client's positiveperceptions of the relationship. Consider the ther-apist behavior of empathy, which is defined byCarkhuff (1971, p. 266) as "the ability to recog-nize, sense, and understand the feelings that an-other person has associated with his behavior andverbal expressions, and to accurately communi-cate this understanding to him." While this defi-nition describes the therapist's expressed empa-thy, it does not address the client's idiosyncraticinterpretation of the therapist's behavior.

In a recent study examining client perceptionsof empathy, Bachelor (1988) found that 44% ofclients perceived their therapist's empathy as cog-nitive, 30% as affective, 18% as sharing, and 7%

as nurturant. Bachelor concluded that empathyhas different meanings to different clients andshould not be viewed or practiced as a univer-sal construct.

The potential for positive enhancement of com-mon factor effects will not occur in those situa-tions in which the therapist's empathic responsedoes not fit the empathic needs of the individualclient. Regardless of how empathic a therapistmay be by the standards of a chosen theoreticalorientation, an empathic response may have littleor no positive impact on certain clients, or maybe interpreted by some clients as having negativeimpact. The therapist's reliance on stand-by re-sponses to convey empathy will not be equallyproductive in terms of the client's perception ofbeing understood (Bachelor, 1988).

Empathy, then, is not an invariant, specifictherapist behavior or attitude (e.g., reflection offeeling is inherently empathic), nor is it a meansto gain a relationship so that the therapist maypromote a particular orientation or personal value,nor a way of teaching clients what a relationshipshould be. Rather, empathy is therapist attitudesand behaviors that place the client's perceptionsand experiences above theoretical content andpersonal values (Duncan, Solovey & Rusk,1992); empathy is manifested by therapist at-tempts to work within the frame of reference ofthe client. When the therapist acts in a way thatdemonstrates consistency with the client's frameof reference, then empathy may be perceived,and common factor effects enhanced. Empathy,therefore, is a function of the client's unique per-ceptions and experience and requires that thera-pists respond flexibly to clients' needs, rather thanfrom a particular theoretical frame of referenceor behavioral set.

Respect

Respect, according to Rogers (1957), is theability to prize or value the client as a person withworth and dignity. Central to conveying respectis a nonjudgmental attitude, the avoidance of con-demnation of the client's actions or motives, andacceptance of the client's experience (Rogers,1957). Demonstrating respect may entail embrac-ing a nonpathological and nonpejorative perspec-tive of people that assumes that all clients canmake more satisfying lives for themselves andhave the inherent capacity to do so. Diagnosticcategories or attributions of pathology that con-note a poor prognosis are perhaps disrespectful

295

B. L. Duncan & D. W. Moynihan

and discount the complexity and beauty of humanvariation, masking the idiosyncratic strengths thatindividuals may utilize to live more satisfyinglives. Such diagnoses may also undermine twoother common factors identified by the Gren-cavage & Norcross' (1990) review, i.e., the cli-ent's positive expectations and therapist qualitiesthat cultivate hope.

Challenging pathology-oriented, diagnosticframes of reference and advocating an abidingfaith in client resources seems unpopular, butwell-founded. Client contribution to outcome(extratherapeutic factors), regardless of diagno-sis, is the single most important factor to success-ful outcome (Lambert, 1992).

Respect is also conveyed by therapists' flexi-bility regarding their interpretations or views ofclients or their circumstances. Interpretationsmade to clients, or therapist views imposed onclients despite the clients' lack of acceptance, aredisrespectful and may undermine common factoreffects. Kuehl, Newfield & Joanning (1990)found that clients who viewed their therapist asnot rigidly adhering to a particular point of viewwere more likely to be satisfied with therapeuticexperiences. Kuehl et al. (1990) conclude thattherapists should proceed cautiously when tryingto convince a client of the utility of an approachthe client does not readily accept. Perceived re-spect may be assured when therapists discard theirapproaches if the client views them as unhelpful.

Respect is demonstrated in therapist attitudesand behaviors that place the value of the clientas a person with worth and dignity above patho-logical, theoretical, or pejorative perspectives; re-spect is manifested by therapist sensitivity to theacceptability of any therapist behavior to the cli-ent's frame of reference (Duncan et al., 1992).

Genuineness

Genuineness means being oneself without be-ing "phony" (Rogers, 1957). Therapists who donot overemphasize their role, authority, or statusare more likely to be perceived as more genuineby clients (Cormier & Cormier, 1991). Genuine-ness may be further operationalized by the thera-pist's cautiousness and tentativeness about ap-proaching the client, conceptualizing his or herconcerns, and intervening to address those con-cerns. Tentativeness conveys to the client that thetherapist claims no corner on reality and is notthe deliverer of truth, but rather is a collaboratorwho, because of training and experience, can

hopefully offer productive input. The therapist,therefore, is not an expert who champions anobjective truth about the etiology and treatmentof client problems or the way life should be lived.Rather, the client is the expert from a perspectivethat places the client's frame of reference andthe client's input in a superior position to thetherapist's orientation or input.

Phoniness may be exemplified by the positionthat equates theories of psychotherapy with"truth", rather than empirical/conceptual approxi-mations of reality. Being genuine with clients maynecessitate a humbling acceptance of the nonde-finitive nature of psychotherapy theory and appli-cation, as well as the inherent complexity of hu-man beings. While theories of psychotherapy areobviously of great value to clinicians and clients,there has been no demonstrated superiority of oneover another. This equivalence of outcome find-ings has been documented in several reviews (Ber-gin & Lambert, 1978; Klein et al., 1983; Orlinsky& Howard, 1986; Luborsky, Singer & Luborsky,1975; Sloane et al., 1975; Smith, Glass & Miller,1980), and more recently in the NIMH multisitestudy of depression (Elkin et al., 1989). Perhapsit is time to reflect such findings in the way clientsare approached.

The failure to find differential outcomes instudies comparing therapies that use highly diver-gent techniques also supports the importance ofcommon factors to positive outcome (Arkowitz,1992; Lambert, 1992). These findings, however,may be interpreted in other ways (Beutler, 1991;Butler & Strupp, 1986; Stiles, Shapiro & Elliott,1986). For example, the apparent equivalence ofoutcome may reflect that different therapies canachieve similar goals through different processes,or that different outcomes do occur but are notdetected by past methodological designs and strat-egies (Arkowitz, 1992; Kazdin & Bass, 1989;Lambert, 1992). The common factors explanationhas received the most attention, and is supportedby other research aimed at discovering the activeingredients of psychotherapy (Lambert, 1992).

Validation

Common factors may also be expressedthrough a therapist's verbal behavior called vali-dation. Validation is a therapist-initiated processin which the client's thoughts, feelings, and be-haviors are accepted, believed, and consideredcompletely understandable, given the client'ssubjective experience of the world. Validation

296

Applying Outcome Research

represents a combined expression of empathy, re-spect, and genuineness that is individually tai-lored to the idiosyncracies of the client's ex-perience.

The therapist genuinely accepts the client's pre-sentation at face value and holds the belief thatthe client is doing the best that he or she can.The therapist respects the client's experience ofthe problem by emphasizing its importance, andempathically offers total justification of the cli-ent's experience. The therapist, therefore, ver-bally legitimizes the client's frame of referenceand in the process may replace the invalidationthat may be a part of it (Duncan et al., 1992).Validation represents a logical application ofcommon factors research as well as studies docu-menting the significance of client perceptions tooutcome.

Client's Frame of Reference: Informal TheoryAnother dimension of the client's frame of ref-

erence encompasses the client's thoughts, beliefs,attitudes, and feelings about the nature of whatserved as the impetus for therapy (the problem orsituation), its causes, and how therapy may bestaddress the client's goals for treatment. Held's(1991) elaboration of the content/process distinc-tion provides a framework for understandingthis dimension.

Content versus Process

Held (1991), building on the work of Prochaska& DiClemente (1982), defines process as the ac-tivities of the therapist that promote change ordevelop coping solutions (i.e., methods, tech-niques, interventions, strategies). Process embod-ies one's theory of how change occurs (Held,1991). Content is the object of the change involv-ing the aspects of the client and his or her behav-ior, upon which the therapist decides to focus theinterventions (Held, 1991).

Content is defined at both formal and informaltheoretical levels (Held, 1991). Formal theoryconsists of either general notions regarding thecause of problems (e.g., symptoms are surfacemanifestations of intrapsychic conflict; symptomsare homeostatic mechanisms regulating a dys-functional subsystem) or predetermined and spe-cific explanatory schemes (e.g., fixated psy-chosexual development; triangulation), whichmust be addressed across cases to solve problems.Cause and effect are either specific or implied byway of theoretical constructs of formal theory.

Those constructs provide the content, which be-come the invariant explanations of the problemsthat bring clients to therapy.

Although variation exists in the degree towhich content is emphasized and elaborated(Held, 1991), most therapies tend to fall to thecontent-oriented pole of the content-processcontinuum. The client presents with a com-plaint, and the therapist will overtly or covertlyrecast the complaint within the language of thetherapist's formal theory. The therapist's re-formulation of the complaint into a specific pre-conceived theoretical content will enable treat-ment to proceed down a particular path flowingfrom the formal theory.

In content-oriented approaches to psychother-apy, the formal theoretical reality of title therapistexists in a hierarchically superior position to theframe of reference of the client. This formal the-ory necessarily structures problem definition aswell as outcome criteria. The more content-ori-ented the approach, the more content-directed thegoals become. Conversely, intentionally utilizingthe client's frame of reference requires that thecontent focus of the therapeutic conversationemerge from the informal theory of the client.

Informal theory involves the specific notionsheld by clients about the nature and causes of theirparticular problems and situations (Held, 1991).Informal theory is revealed through clients' artic-ulations and elaborations of their concerns and isnecessarily highly idiosyncratic. Recall the Bach-elor (1991) study, which indicated the importanceof not only the therapist-provided variables, butalso the therapist's efforts to explore material thatthe client perceived as relevant. Clients seem towant therapists to explore their informal theories.

Rather than reformulating the informal theoryinto the language of the therapist's formal theory,it is suggested that therapists accommodate theirformal theories to the client's informal theory byelevating the client's perceptions and experiencesabove theoretical conceptualizations, therebyallowing the client's informal theory to dictatetherapeutic choices. Understanding the client'ssubjective experience and phenomenological rep-resentation of the presenting problem, and placingthat experience above the theoretical predilectionof the therapist seems consistent with the notionof enhancing common factor effects. Adoptingthe client's informal theory also provides a sig-nificant step in securing a strong therapeuticalliance.

297

B. L. Duncan & D. W. Moynihan

The Informal Theory and the TherapeuticAlliance

Such an accommodation to the client's infor-mal theory appears warranted given the impor-tance of client perceptions to outcome (Gurman,1977; Bachelor, 1991; Horowitz et al., 1984) andthe large body of evidence demonstrating that thetherapeutic alliance, as rated by client, therapist,and third-party perspectives, is the best predictorof psychotherapy outcome (Alexander & Lubor-sky, 1986; Marmar et al., 1986; Marziali, 1984;Suh, Strupp & O'Malley, 1986).

Bordin (1979) formulated three interactingcomponents of the alliance: 1) agreement on thegoals of psychotherapy; 2) agreement on the tasksof psychotherapy (specific techniques, topics ofconversation, interview procedures, frequency ofmeeting); and 3) the development of a relationshipbond between the therapist and client. While thebonding dimension reiterates the importance ofthe relationship and the therapist-provided vari-ables, the agreement on goals and tasks refersto the congruence between the client's and thetherapist's beliefs about how people change intherapy (Gaston, 1990).

Adopting the client's informal theory may en-sure the development of a strong therapeutic alli-ance. By allowing the client's idiosyncratic con-tent focus to direct the therapeutic process, thereis necessarily an agreement regarding goals andtasks because the therapist always accommodatesformal theory(s) to the informal theory of theclient. The therapist attends to what the clientthinks is important, addresses what the client indi-cates as significant, and accommodates both in-and out-of-session intervention to accomplishgoals specified by the client.

Each client, therefore, presents the therapistwith a new theory to learn and a different thera-peutic course to pursue. Emerging from the pro-cess of unfolding the client's frame of refer-ence, the therapist, an active participant, drawsupon theoretical frames of reference and addsinput, leading to the evolution of a new theoryor frame of reference. Clients reconceptualizetheir informal theories by combining aspects oftheir experience with alternative views thatarise from therapeutic dialogues. The alterna-tive views are only perspectives that the clientachieves in the process. Psychotherapy can beconceptualized as an idiosyncratic, process-de-termined synthesis of ideas, formulated by theclient, that culminates in a new theory with ex-

planatory and predictive validity for the client'sspecific circumstance.

Intervention and Common Factor EffectsAlthough it is useful to examine common fac-

tors separate from specific technique, relationshipand intervention factors are interdependent as-pects of the same process. Butler and Strupp(1986) argue:

The complexity and subtlety of psychotherapeutic processcannot be reduced to a set of disembodied techniques, becausethe techniques gain their meaning, and in turn, their effective-ness, from the particular interaction of the individualsinvolved (p. 33).

The interactional context that creates meaningfor intervention are the characteristics, attitudes,and behaviors of the therapist that provide thecore conditions as perceived by the client. Com-mon factors may be enhanced by specific inter-ventions that convey or implement the therapist'sunderstanding and acceptance of, as well as re-spect for, the client's frame of reference. Inter-vention, then, becomes another behavioral mani-festation of the relationship. Intervention in theform of tasks or assignments extends the interper-sonal context defined in session to the client'ssocial environment and offers another opportunityto enhance common factor effects and maximizea positive therapeutic alliance.

The expert therapist role is therefore de-empha-sized in the current proposal. From an expert posi-tion, the therapeutic search is for interventionsreflecting objective truths that promote change viathe process of validating the therapist's theoreticalpoint of view. The therapeutic search, from aposition seeking to deliberately influence success-ful outcome, is for interventions reflecting subjec-tive truths that promote change via the process ofvalidating the client's frame of reference. Threegeneral steps extend the common factors andstrong alliance context to the intervention process.

Dependence on the Client's Resources

People who enter therapy, except in certaincompulsory situations, do so because the experi-ence of their lives or some specific circumstancehas become so painful that a change of some kindis perceived as necessary. Clients may initiallyappear frustrated and helpless, creative energiesmay be at a low ebb, and the perception mayexist that they have tried everything possible onlyto have experience failure time after time. Theclient's frustration and helplessness should be re-

298

Applying Outcome Research

spected by the therapist and not considered as areflection of any deficits or psychopathology.

Such a perspective is critical because the thera-pist is counting on the existent resources andstrengths of the client. Interventions offer oppor-tunities for change that promote clients' utiliza-tion of their own inherent capacities for growth.A pathology perspective may undermine the ther-apist's confidence in the resources of clients and,therefore, limit the range of interventions fromwhich to choose. An initial step, then, is the rec-ognition that interventions depend upon the re-sources of clients for success. Therapist depen-dence may perhaps hierarchically align thetherapeutic relationship in a way that promotesthe alliance and enhances relationship effects.

What the Client Wants

The next general step is that intervention mustaddress what the client defines as problematic andwhat the client indicates as the goal for therapy.Rather than an imposition of the therapist's theo-retical (or personal) frame of reference, the inter-vention is a response to the client's formulationof the problem experience. The client's desires,therefore, set the focus and structure of the inter-vention process.

Intervention begins by accepting the client'spresentation at face value, without any reformula-tion, and then accommodating the intervention tothat general presentation of the problem situationvia rationales and treatment options available tothe therapist. This aspect of intervention is limitedonly by the therapist's resources and knowledgebase. The second step, then, is characterized byan explicit therapist acceptance of what the clientwants and a start of a general search for interven-tion options that directly address the client'sdesires.

Collaborative Exploration

The final step involves the recognition that in-tervention is a collaborative exploration processthat emerges from the therapist-client conversa-tion and the clients' articulation of their content-rich frame of reference. As clients tell their prob-lem stories, they elaborate the idiosyncracies oftheir experiences, their views of the problem itselfand perhaps how it may be best approached, andwhat they have tried to do previously to solvethe problem.

The client, then, collaborates in the interven-tion process during the interview by virtue of his

or her description of, and dialogue about, theproblem experience. The therapist continuouslyevaluates the multiple options and begins to ruleout choices that are obviously antithetical to theclient's informal theory. Differential therapeuticsgleaned from the literature serve as a guide tothe intervention process. However, the therapistdepends upon the client's receptivity to ideas gen-erated regarding views and actions about the cli-ent's concerns. Interventions evolving from col-laborative exploration demonstrate the therapist'sacceptance and validation of the client's frame ofreference each time the client enacts the interven-tion. The common factors context of the relation-ship is therefore extended outside the session tothe client's social environment. Out-of-sessionvalidation may encourage clients to utilize theirresources to resolve problems.

Discussion and ConclusionRecall the percentages of outcome variance at-

tributable to four therapeutic factors: extrathera-peutic change accounts for 40% of outcome vari-ance, common factors account for 30%, whileplacebo and specific technique each contribute to15% of the variance (Lambert et al., 1986). Thedifferential percentages of the four factors reflecttheir differential emphasis in the current proposal.

Client-specific variables that result in changespeak to clients' inherent resources, as well as theirability to utilize out-of-therapy events (e.g., socialsupport, fortuitous events) as opportunities forchange. Given that this factor is percentage-wisethe most powerful, the therapeutic process may beviewed as empowering a context that enables clientsto access their own capacities for growth. From thisperspective, intervention attempts to create opportu-nities for extratherapeutic change.

This article proposed to maximize the effectsof therapist-provided variables by intentionallyutilizing the client's frame of reference (experi-ence of the relationship and informal theory) andextending a strong alliance into the interventionprocess by depending on the client's resources,addressing what the client wants, and collabora-tively exploring intervention options. It was sug-gested that intervention represents another behav-ioral manifestation of the therapeutic relationshipthat offers the opportunity for clients to experi-ence validation of their frame of reference eachtime they enact the intervention.

Last place in terms of significance to outcomeis shared by placebo and specific technique fac-

299

B. L. Duncan & D. W. Moynihan

tors. Placebo or expectancy effects include im-provement that results from the client's knowl-edge of being in treatment, and consists ofvariables such as therapist credibility and the useof encouragement, persuasion, and reassurance(Lambert, 1992). The selection of the content forconservation, as well as technique based upon theclient's frame of reference, explicitly addressesand, therefore, enhances client expectancies re-garding therapy. Meeting the client's expectationsregarding the goals and tasks of therapy wouldappear to enhance placebo effects by creating acognitive set which expects change. Discussionof the applicability of a particular interpretationor intervention with a client may empower pla-cebo by enhancing the intervention's credibility,as well as conveying the therapist's encourage-ment and reassurance. Expectancy may also beenhanced and hope cultivated by the therapist'scognitive set which conveys that change is inevi-table given the client's resources and abilities.

Since technique only represents 15% of out-come variance, techniques may be viewed onlyas formal content areas that may or may not proveuseful in the unique circumstance of the client.The selection of technique or content, therefore,must go beyond the mere prescriptive matchingof client problems with research-demonstratedtechniques. It seems that outcome depends farmore on the client's resources and enactment ofthe technique, the therapist's style, attitude, andinterpersonal relationship with the client, and thecongruence of the technique with the client'sframe of reference.

Biased by three decades of investigation ofthe factors accounting for successful outcomeand the recent findings regarding the therapeuticalliance, this article suggested a more inten-tional utilization of the client's frame of refer-ence and a deliberate promotion of a client-directed process in psychotherapy. Empoweringextratherapeutic change, enhancing commonfactor effects, and building a strong alliance arenot passive therapist postures, but rather areproactive initiatives that require a planned, fo-cused effort to conduct psychotherapy withinthe context of the client's frame of reference.The outcome research suggests that psychother-apy devote itself more to a process directed bythe individual client's construction of what con-stitutes success in therapy. Such a client-di-rected process may only enhance the value ofempirically demonstrated differential therapeutics.

ReferencesALEXANDER, L. B. & LUBORSKY, L. (1986). The Penn helping

alliance scales. In L. S. Greenberg and W. M. Pinsof(Eds.), The psychotherapeutic process: A research hand-book (pp. 325-366). New York: Guilford.

ARKOWTTZ, H. (1992). A common factors therapy for depres-sion. In J. C. Norcross and M. R. Goldfried (Eds.), Hand-book of psychotherapy integration (pp. 402-432). NewYork: Basic.

BACHELOR, A. (1988). How clients perceive therapist empa-thy. Psychotherapy, 25, 227-240.

BACHELOR, A. (1991). Comparison and relationship to out-come of diverse dimensions of the helping alliance as seenby client and therapist. Psychotherapy, 28, 534-549.

BERGIN, A. E. & LAMBERT, J. J. (1978). The evaluation ofoutcomes in psychotherapy. In S. L. Garfield and A. E.Bergin (Eds.), Handbook of psychotherapy and behaviorchange: An empirical analysis (pp. 139-189). New York:John Wiley.

BEUTLER, L. E. (1991). Have all won and must all haveprizes? Revisiting Luborsky et al.'s verdict. Journal of Con-sulting and Clinical Psychology, 59, 226-237.

BORDIN, E. S. (1979). The generalizability of the psychoana-lytic concept of the working alliance. Psychotherapy, 16,252-260.

BUTLER, S. F. & STRUPP, H. H. (1986). Specific and nonspe-cific factors in psychotherapy: A problematic paradigm forpsychotherapy research. Psychotherapy, 23, 30-40.

CARKHUFF, R. R. (1971). The development of human re-sources. New York: Holt, Rinehart & Winston.

CORMIER, W. H. & CORMIER, L. S. (1991). Interviewing strate-gies for helpers (3rd Ed.). Pacific Grove, CA: Brooks/Cole.

DUNCAN, B., SOLOVEY, A. & RUSK, G. (1992). Changing

the rules: A client-directed approach to therapy. NewYork: Guilford.

ELKIN, I., SHEA, T., WATKINS, J. T., IMBER, S. D., SOTSKY,S. M., COLLINS, I. F., GLASS, D. R., PILKONIS, P. A.,LEBER, W. R., DOCKERTY, J. P., FIESTER, S. J. & PARLOFF,M. B. (1989). National Institute of Mental Health treatmentof depression collaborative research program: General ef-fectiveness of treatments. Archives of General Psychiatry,46, 971-982.

GASTON, L. (1990). The concept of the alliance and its rolein psychotherapy: Theoretical and empirical considerations.Psychotherapy, 27, 143-152.

GRENCAVAGE, L. M. & NORCROSS, J. D. (1990). Where arethe commonalities among the therapeutic common factors?Professional Psychotherapy: Research and Practice, 21,372-378.

GURMAN, A. S. (1977). Therapist and patient factors influ-encing the patient's perception of facilitative therapeuticconditions. Psychiatry, 40, 16-24.

HELD, B. S. (1991). The process/content distinction in psy-chotherapy revisited. Psychotherapy, 28, 207-217.

HOROWITZ, M., MARMAR, C , WEISS, D., DEWITT, K. &ROSENBAUM, R. (1984). Brief psychotherapy of bereave-ment reactions: The relationship of process to outcome.Archives of General Psychiatry, 41, 438-448.

KAZDIN, A. E. &BASS, D. (1989). Power to detect differencesbetween alternative treatments in comparative psychother-apy outcome research. Journal of Consulting and ClinicalPsychology, 57, 138-147.

KLEIN, D., ZTTRIN, C , WOERNER, M. & Ross, D. (1983).Treatment of phobias: II. Behavior therapy and supportive

300

Applying Outcome Research

psychotherapy: Are there any specific ingredients? Archivesof General Psychiatry, 40, 139-145.

KUEHL, B. P., NEWHELD, N. A. & JOANNING, H. (1990). Aclient-based description of family therapy. Journal of Fam-ily Psychology, 3, 310-321.

LAMBERT, M. (1992). Psychotherapy outcome research. InJ. C. Norcross and M. R. Goldfried (Eds.), Handbook ofpsychotherapy integration (pp. 94-129). New York: Basic.

LAMBERT, M. J., SHAPIRO, D. A. & BERGIN, A. E. (1986).The effectiveness of psychotherapy. In S. L. Garfield andA. E. Bergin (Eds.), Handbook of psychotherapy and be-havior change (3rd ed., pp. 157-212). New York: JohnWiley.

LUBORSKY, L., SINGER, B. & LUBORSKY, L. (1975). Compara-tive studies of psychotherapies: Is it true that "everybodyhas won and all must have prizes"? Archives of GeneralPsychiatry, 32, 995-1008.

MARMAR, C., HOROWITZ, M., WEISS, D. & MARZIALI, E.(1986). The development of the Therapeutic Alliance Rat-ing System. In L. Greenberg and W. Pinsof (Eds.), Thepsychotherapeutic process: A research handbook (pp. 367-390). New York: Guilford.

MARZIALI, E. (1984). Three viewpoints on the therapeuticalliance: Similarities, differences, and associations withpsychotherapy outcome. Journal of Nervous and MentalDisease, 172, 417-423.

ORUNSKY, D. E. & HOWARD, K. I. (1986). Process and out-come in psychotherapy. In S. L. Garfield and A. E. Bergin

(Eds.), Handbook of psychotherapy and behavior change(3rd ed., pp. 311-381). New York: John Wiley.

PATTERSON, C. H. (1984). Empathy, warmth, and genuinenessin psychotherapy: A review of reviews. Psychotherapy, 21,431-438.

PATTERSON, C. H. (1989). Foundations for a systematic eclec-tic psychotherapy. Psychotherapy, 26, 427-435.

PROCHASKA, J. O. & DICLEMENTE, C. C. (1982). Transtheo-retical therapy: Toward a more integrative model of change.Psychotherapy 19, 276-288.

ROGERS, C. R. (1957). The necessary and sufficient conditionsof therapeutic personality change. Journal of ConsultingPsychology, 21, 95-103.

SLOANE, R. B., STAPLES, F. R., CRISTOL, A. H., YORKSTON,N. J. & WHIPPLE, K. (1975). Psychotherapy versus behav-ior therapy. Cambridge, MA: Harvard University Press.

SMITH, M. L., GLASS, G. U. & MILLER, T. J. (1980). The

benefits of psychotherapy. Baltimore: Johns Hopkins Uni-versity Press.

STILES, W. G., SHAPIRO, D. A. & ELLIOTT, R. (1986). "Areall psychotherapies equivalent?" American Psychologist,41, 1-8.

SUH, C , STRUPP, H. & O'MALLEY, S. (1986). The Vanderbiltprocess measures: The Psychotherapy Process Scale(VPPS) and the Negative Indicators Scale (VNIS). In L.Greenberg and W. Pinsof (Eds.), The psychotherapeuticprocess: A research handbook, (pp. 285-323). NewYork: Guilford.

301