Slone et al. 2105

Evaluating the Efficacy of Client Feedback in Group Psychotherapy

Norah C. SloneSpalding University

Robert J. ReeseUniversity of Kentucky

Susan Mathews-DuvallLexington, Kentucky

Jonathan KodetUniversity of Kentucky

Monitoring progress in psychotherapy routinely (i.e., client feedback) has yieldedpositive results for improving both outcome and retention in individual and coupletherapy; however, evidence of client feedback efficacy in a group format is limited.Using a randomized cluster design, group therapy participants (N � 84) were assignedto a client feedback or treatment-as-usual (TAU) condition in a university counselingcenter. Clients in the feedback condition used the Partners for Change OutcomesManagement System (PCOMS; Duncan, 2011). Feedback participants had significantlylarger pre–post group therapy gains (d � 0.41) and higher rates of reliable andclinically significant change when compared to TAU participants on the OutcomeRating Scale (Miller & Duncan, 2000). Clients in the feedback condition also attendedmore group sessions compared to TAU participants. Study implications and futurerecommendations are provided.

Keywords: group psychotherapy, client feedback, process and outcome, continuous assessment,treatment monitoring

Group therapy has consistently been found tobe an effective form of therapy, and generallyfound to have comparable treatment outcomesto individual therapy for a variety of presentingissues (Burlingame, Strauss, & Joyce, 2013;McRoberts, Burlingame, & Hoag, 1998). In ad-dition to being similarly effective with regard toclient outcome, individual and group treatmentformats have two less encouraging factors incommon. First, a sizable number of clients donot benefit from treatment. The percentages ofclients who terminate prematurely (i.e., drop-ping out therapy before goals are achieved) or

deteriorate in treatment seem to be comparableacross formats (Swift, & Greenberg, 2012; Wi-erzbicki & Pekarik, 1993). Although the grouptherapy literature lags behind in formally esti-mating dropout across studies, estimates fromindividual therapy have historically reflectedthat approximately 33–50% of clients drop outof therapy and deteriorate at rates between 5 and10% (Lambert & Ogles, 2004). More recentestimates suggest that dropout in individualtherapy is approximately 18.7%, whereas drop-out in group treatment is approximately 19.7%(Swift & Greenberg, 2012).

Second, clinicians in both treatment formatshave difficulty identifying which clients are notbenefiting from treatment (Chapman et al.,2012; Hannan et al., 2005). Chapman and col-leagues replicated results of a study evaluatingclinical prediction in individual therapy withgroup members (see Hannan et al., 2005, fordetails). They found that therapists blinded tooutcome scores on the Outcome Question-naire-45 (OQ-45; Lambert et al., 1996) haddifficulty accurately predicting clinical out-comes for their group clients (N � 64; only 49

Norah C. Slone, School of Professional Psychology,Spalding University; Robert J. Reese, Department of Edu-cational, School, and Counseling Psychology, University ofKentucky; Susan Mathews-Duvall, Private Practice, Lex-ington, Kentucky; Jonathan Kodet, Department of Educa-tional, School, and Counseling Psychology, University ofKentucky.

Correspondence concerning this article should be ad-dressed to Norah C. Sloane, School of Professional Psy-chology, Spalding University, 845 South Third Street, Lou-isville, KY 40067. E-mail: [email protected]

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Group Dynamics: Theory, Research, and Practice © 2015 American Psychological Association2015, Vol. 19, No. 2, 000 1089-2699/15/$12.00 http://dx.doi.org/10.1037/gdn0000026

1

completed prepost data) from either a universitycounseling center (n � 33) or inpatient psychi-atric hospital (n � 31). Specifically, therapistspredicted that 31 of 49 clients would demon-strate reliable improvement on the OQ-45.However, only 13% (or four out of 31 clients)of their predictions were accurate. No therapistaccurately predicted clients who reliably deteri-orated (n � 10).

Client feedback, or monitoring client out-come throughout treatment, was developed spe-cifically to address these two concerns that af-fect client outcome across treatment formats.Monitoring progress in treatment assists clini-cians with identifying clients who are at-risk fordropping out due to a lack of improvement orworsening in therapy and offers the opportunityto alter or amend treatment in a manner thatmay better suit the client. Results have gener-ally indicated that client feedback decreasespremature termination rates and improves gen-eral psychotherapy outcomes for both individu-als (Lambert et al., 2001, 2002; Reese, Nor-sworthy, & Rowlands, 2009; Reese, Duncan,Bohanske, Owen, & Minami, 2014) and couples(Anker, Duncan, & Sparks, 2009; Reese, To-land, Slone, & Norsworthy, 2010). Less re-search exists with groups. Research that doesexist offers mixed results (Davies, Burlingame,Johnson, Gleave, & Barlow, 2008; Schuman,Slone, Reese, & Duncan, 2014), providing anunclear picture as to whether benefits of clientfeedback extend to group psychotherapy.

Although there are several formal feedbacksystems (see Castonguay, Barkham, Lutz, &McAleavey, 2013), only two have been evalu-ated using a randomized clinical trial (RCT)design in the United States: the Outcome Ques-tionnaire System (OQ System; Lambert, Han-son, & Harmon, 2010) and the PCOMS (Dun-can, 2011, 2012, 2014). Both feedback systemsare included in the Substance Abuse and MentalHealth Administration’s (SAMHSA) NationalRegistry of Evidence-based Programs and Prac-tices. The OQ System uses the OQ-45, a 45-item measure of global distress, to monitortreatment progress and identify clients who arenot-on-track (NOT; client at-risk for negativeoutcome or premature termination). If identi-fied, clinical support tools that measure the ther-apeutic alliance, readiness for client change, andlevel of social support are used to further eval-uate the lack of treatment progress. Evidence

for the efficacy of the OQ System is based onnine RCT studies that all show significant treat-ment gains for NOT clients who were in afeedback condition compared to treatment asusual (TAU; Crits-Christoph et al., 2012; Har-mon et al., 2007; Hawkins, Lambert, Ver-meersch, Slade, & Tuttle, 2004; Lambert et al.,2001; Lambert et al., 2002; Probst et al., 2013;Slade, Lambert, Harmon, Smart, & Bailey,2008; Simon et al., 2013; Whipple et al., 2003).Shimokawa, Lambert, and Smart (2010) con-ducted a meta-analysis with six of the earliestRCT studies (N � 6,151) and found clients inthe feedback conditions had approximately 2.6times higher odds of attaining reliable improve-ment compared to clients in the TAU condi-tions.

The other client feedback system, PCOMS,has also demonstrated significant treatmentgains for feedback over TAU in three RCTs(Anker, Duncan, & Sparks, 2009; Reese, Nor-sworthy, & Rowlands, 2009; Reese, Toland,Slone, & Norsworthy, 2010). PCOMS usesmuch shorter measures, consisting of three brieffour-item instruments, the Outcome RatingScale (ORS; Miller & Duncan, 2000), the Ses-sion Rating Scale (SRS; Miller, Duncan, &Johnson, 2000) for individual and couple ther-apy, and the Group Session Rating Scale(GSRS; Duncan & Miller, 2007) for group ther-apy. The ORS is administered at the beginningof every session to monitor treatment progressand the SRS or GSRS are administered at theend every session to monitor the therapeuticalliance. In individual therapy, Reese et al.(2009) found significant treatment gains forfeedback when compared to TAU in both auniversity counseling center (N � 74) and agraduate training clinic (N � 74). Moreover,those in the PCOMS conditions achieved reli-able change in significantly fewer sessions thanTAU. Two studies on couples therapy havecomparable results. In Anker et al. (2009), 205couples randomized to PCOMS or TAUshowed statistically and clinically significantimprovements for couples who provided feed-back. Couples in the feedback conditionachieved clinically significant change approxi-mately four times more than those in the TAUcondition. Moreover, these changes were main-tained at 6-month follow-up with significantlymore couples in the feedback condition remain-ing together. The findings from the Anker et al.

2 SLONE, REESE, MATHEWS-DUVALL, AND KODET

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

study were replicated by Reese et al. (2010), butwith fewer couples (N � 46 couples).

In addition to demonstrating improvementsin outcome, client feedback has been shown toimprove deterioration and attendance rates. Forexample, Lambert and Shimokawa’s (2011)meta-analytic results suggested that clients in afeedback condition using either PCOMS or theOQ System experienced less than half odds ofdeteriorating in treatment; however, results withthe OQ System only applied to clients not ontrack for positive outcomes, or NOT. Feedbackalso improves attendance rates, though resultsare mixed as to whether all clients benefit or justthose NOT. Specifically, Lambert et al. (2001)found that clients who were NOT in a feedbackcondition attended significantly more sessions(9.68 vs. 5.03) than NOT clients in a TAUcondition. Conversely, Slade et al. (2008) foundthat all clients, not just those NOT, in a feed-back condition with clinical support tools at-tended 1.5 more sessions compared to TAU.

Client feedback with individual and coupletherapy has been found to be efficacious, but theresults have not been replicated in the groupformat. The use of client feedback in grouptherapy is worthy of investigation given thesame concerns exist as in individual therapy,namely with regards to premature terminationand the inability of the therapist to identifyclients at-risk for not benefitting in therapy.Formally monitoring treatment progress in agroup format may prove critical given that it canbe difficult to assess individual client progresswithin a group using only clinical judgment.Given that we already know clinicians havedifficulty identifying clients not progressing ingroups on average (Chapman et al., 2012), for-mally monitoring client progress routinely mayhelp avoid such clients getting “lost” in the mix.

We could only identify two studies evaluat-ing the efficacy of client feedback in grouppsychotherapy (Davies, Burlingame, Johnson,Gleave, & Barlow, 2008; Schuman, Slone, Re-ese, & Duncan, 2014). Davies et al. (2008)studied the effects of a feedback interventionacross a mean number of six group sessionsusing the OQ-45 as an outcome measure atpre–post and the Group Climate Questionnaire-Short version (GCQ-S; MacKenzie, 1983) atevery session to monitor group member’s rela-tionships. Feedback on the group climate wasgiven in narrative and graphical (i.e., visual

graph of GCQ-S subscale scores) forms to cli-ents (N � 94), coleaders, and the group-as-a-whole at a university counseling center. Re-searchers found that monitoring group climatedid not enhance client outcome or level of en-gagement in the group compared to a TAUcondition. In fact, client feedback using theGCQ-S only significantly predicted worse out-comes for clients who rated their group as ex-periencing higher levels of conflict overall.

Second, Schuman et al. (2014) evaluated cli-ent feedback using a limited PCOMS interven-tion (i.e., excluded the GSRS) in a group ther-apy format with active military personnel in theArmy accessing services for substance abusetreatment. Clients in a feedback condition dem-onstrated significantly more pre–post treatmentgains (d � 0.28) and attended more sessionscompared to a TAU group therapy condition. Inaddition, more clients who were NOT in thefeedback condition were retained. Lastly, cli-ents in the feedback condition achieved signif-icantly higher satisfaction ratings from theirtherapist and commanding officers in compari-son to the TAU condition. Although resultswere positive for this study, two primary limi-tations exist: only five group sessions were mea-sured and group cohesion was not monitored asthe GSRS was not developed at the time thisstudy was implemented.

The purpose of this study was to further eval-uate the efficacy of client feedback in grouppsychotherapy, addressing both the lack of stud-ies and the limitations noted in the previousclient feedback group psychotherapy studies.Specifically, we evaluated whether using a cli-ent feedback system enhanced treatment out-come and retention when compared to TAUgroup therapy. We had three hypotheses. First,we hypothesized that clients in a feedback con-dition would have significantly larger pre–postpsychotherapy outcome gains. Second, we hy-pothesized that clients in a feedback conditionwould achieve significantly higher rates of reli-able and clinically significant change. Third, wehypothesized that clients in a feedback condi-tion would attend more sessions and have lowerrates of premature termination. Although boththe OQ System and PCOMS have been evalu-ated in the group studies above, PCOMS waschosen for the current study given its brevityand the availability of a group alliance measure

3EVALUATING FEEDBACK IN GROUP THERAPY

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

and protocol were available (Duncan, 2011;Duncan & Sparks, 2010).

Methods

Participants

A total of 90 participants attended group ther-apy at a large university counseling center in theSoutheast from January 2012 to December2012. However, five participants declined toparticipate in the research study and one partic-ipant was dropped from the analysis becauseshe did not complete the measures as instructed(e.g., made erratic marks later determined to beunrepresentative of the client’s current experi-ence). Participants (N � 84) were clients whotook part in interpersonal process group therapyfor social anxiety and interpersonal concerns. Atotal of 43 clients participated in a feedbackcondition and 41 were in the TAU condition.All but two participants were new group therapyclients who attended a group for one academicsemester. The two clients attended group fortwo academic semesters, first participating in aTAU condition then a feedback condition.

Client participants had a mean age of 21.5(SD � 2.7; range 18–28), which was similaracross treatment conditions (feedback M �21.5, TAU M � 21.5), identified as mostlywomen (64.3%), and Caucasian (84.5%; 10.7%identified as African American/Black, 1.1% asmultiracial, and 1.1% asked to self-identify).Most reported their sexual identity as heterosex-ual (85.7%), though some identified as gay(4.8%), lesbian (3.6%), bisexual (3.6%), orquestioning (2.4%). Clients reported beingmostly single (66.7%), though some were in aserious or dating or committed relationship(27.4%), married (3.6%), or divorced (2.4%).Clients were classified as freshmen (23.8%),sophomores (14.3%), juniors (21.4%), seniors(17.9%), and graduate or professional degreestudents (22.6%). Approximately 58.3% of thesample endorsed attending individual therapyprior to their group experience, though the re-cency of their treatment is not known. Althoughclients’ specific presenting concerns or diagno-ses were not monitored in this study, clients atthis center endorsed the following top threepresenting concerns in order of highest percent-age endorsed (more than one could be checked):anxiety (68%), stress (64%), and depression

(58%). The use of psychotropic medication wasalso not monitored in this study.

Group Leaders

Graduate students, predoctoral interns, anddoctoral-level staff psychologists (N � 20)served as group leaders. Two leaders were as-signed to each group, one who had more groupexperience (e.g., senior staff member or expe-rienced predoctoral intern), with a graduate stu-dent trainee in a clinical or counseling psychol-ogy doctoral program, predoctoral intern, orless experienced (�3 years of experience withinterpersonal process groups) senior staff mem-ber. The Group Psychotherapy Coordinator, alicensed doctoral-level staff psychologist at thecounseling center, determined pairs of coleadersat the outset of each semester prior to groupsbeing randomized for the study. The Coordina-tor also provided weekly group supervision for1.5 hr with all group leaders in both conditions.All group leaders received the same amount ofsupervision from the same supervisor through-out the duration of this study. Group leaders inboth conditions were part of the same groupsupervision session; separating coleaders wasnot modifiable by research personnel at the site.However, the only discussion of the study thattook place in the combined supervision sessionwas related to treatment fidelity. No other con-versation about the process of using the studymeasures in group therapy sessions was dis-cussed during group supervision. Coleaders inthe feedback condition were prompted to con-sult about specific concerns related to the feed-back protocol (i.e., how to respond to signalalarm system with group members in session)with study personnel outside of group supervi-sion.

Group leaders were a mean age of 32.1 years(SD � 8.2; range of 23–52), most identified aswomen (80%) and Caucasian (85.0%; Asian/Pacific Islander, 10.0%; and multiracial, 5.0%).Eight group leaders had doctoral degrees, and12 were doctoral student trainees. Group leadersreported an average of 4.8 years of group psy-chotherapy experience (SD � 7.1; range 0–22years) and had led an average of 4.5 groups(SD � 4.4; range: 0–15) prior to this study.They identified their general theoretical orien-tations as mostly integrative (65.0%), thoughsome identified primarily with interpersonal


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

process therapy (20.0%) and cognitive behav-ioral therapy (15.0%) orientations. No signifi-cant differences were noted on demographicvariables, including experience level, acrosstreatment conditions (p � .05).

Measures

ORS. The ORS is a brief four-item measureof global psychological functioning that was de-signed for use every session to monitor clientoutcome (Duncan, 2011; measures may be down-loaded for free at https://heartandsoulofchange.com). Four domain scores derived from subscalesof the OQ-45, including Individually (personalwellbeing), Interpersonally (family and close re-lationships), Socially (school, work, and otherfriendships), and Overall (general measure of psy-chological wellbeing), measure global psycholog-ical functioning using a visual analog scale 10 cmin length for each area assessed. Although anelectronic version exists, the paper-based versionof the measure was used. To administer this in-strument, therapists ask clients to place a hashmark on the line that best represents how theywere feeling over the past week with scores on theright side of the scale indicating higher function-ing. A ruler is used to score each domain to thenearest millimeter, with 10 being the highest scoreobtainable for each domain. Total scores rangefrom 0 to 40. The total score is plotted on a grapheach session to track their progress throughouttreatment. The ORS can be administered, scored,plotted, and interpreted in approximately one min-ute.

The protocol procedures for clients respond-ing to the ORS in a group format were nuancedfor logistical purposes. When clients entered thegroup session each week, they were instructedto complete, score, and plot the total scores ontheir individual graph. When timed for researchpurposes, this process took no more than 2.5minutes. Coleaders prompted clients to sharetheir results through check in, which was inter-woven into the process of group therapy. Giventhis, it is difficult to state exactly the length oftime members spent sharing their ORS resultsas it was included into their routine check in andgroup process.

Several studies provide empirical evidencefor the reliability and validity of scores gener-ated by the ORS (e.g., Bringhurst, Watson,Miller, & Duncan, 2006; Campbell & Hemsley,

2009; Duncan, Sparks, Miller, Bohanske, &Claud, 2006; Gillaspy & Murphy, 2011; Miller,Duncan, Brown, Sparks, & Claud, 2003). Anaverage Cronbach’s coefficient alpha for allstudies was .85. Test-retest coefficients havebeen found to range from r � .66 (Miller et al.,2003) after 1–3 weeks to r � .80 (Bringhurst etal., 2006) after the same time period when ORStotal scores have been used. Internal consis-tency estimates using ORS total scores for thecurrent sample was .92 across 327 total admin-istrations.

Concurrent validity was also estimated com-paring the ORS and OQ-45. Miller et al. (2003)paired 335 administrations of the ORS and theOQ-45 across four sessions using a nonclinicalsample (N � 86) and found a correlation of r �.59. Bringhurst et al. (2006) reported a strongerrelationship (r � .69) at the third administration(range of r � .56–.57 across first two adminis-trations). Both studies suggest moderate evi-dence supporting the concurrent validity of theORS as a measure of global psychological func-tioning.

To facilitate clinical use of the ORS, Jacob-son and Truax’s (1991) formulas were used todetermine the ORS clinical cutoff and the reli-able change index (RCI) for evaluating clini-cally significant change. Miller et al. (2003)used a nonclinical, community sample (n � 86)and a clinical sample (n � 435) to establish acut score of 25. The RCI for the ORS wascomputed using 34,790 participants and wasdetermined to be 5 points (Duncan, 2011). Toachieve clinically significant change, a clientmust begin treatment with an ORS score �25,improve by at least 5 points, and finish treat-ment with an ORS score �25.

The GSRS, a group cohesion measure (i.e.,member’s relationship to the group, includingleaders, rather than just the relationship betweenclient and therapist), was utilized with the feed-back group only, but it is not fully reviewedhere given it is used as part of the client feed-back system intervention itself and not includedin the data analysis separately. The GSRS is afour-item measure that uses a visual analogscale, that assesses four domain areas related toBordin’s (1979) theory of the working alliance:relationship (i.e., I felt understood and respectedby the leader and the group), goals and topics(i.e., We worked and talked about what i wantedto and talk about), approach or method (i.e., The


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

leader and the group’s approach is a good fit forme), and overall (i.e., Today’s group was rightfor me). It was administered, scored, and inter-preted in the same manner as the ORS at the endof group session.

Procedures

We used a cluster RCT design. Randomiza-tion occurred at the group level, in whichgroups-as-a-whole were randomly assigned to afeedback or TAU condition. Randomizing cli-ents and coleaders individually to groups wasnot feasible due to client, staff, and group lead-ers’ schedules, as they were only available atcertain times.

Groups (N � 10; feedback � 5 and TAU �5) were closed (i.e., more members were notadded once formed) interpersonal process psy-chotherapy groups (Yalom, 1995). Groups atthe center are designed to last one academicsemester. For the current study, the groups con-ducted an average of 10 weekly sessions (10.4in the feedback condition and 9.6 in the TAUcondition) for 1.5 hours each week. Groupsstarted over each semester with new membersand coleaders. Each group was comprised of 8.5clients on average (8.8 in feedback condition[SD � 1.10]; 8.2 in TAU condition [SD �1.30]). The group coordinator, a doctoral-levelstaff psychologist at the center, made grouptimes and coleadership assignments at the be-ginning of each semester. These aspects werenot modifiable by research personnel.

Group psychotherapy clients at this centerwere recruited through referral procedures andparticipated according to inclusion/exclusioncriteria outlined by center policies. Clients whopresented with concerns related to social anxi-ety or other interpersonal issues were referred togroup therapy by their intake or individual ther-apist, although some clients presented withother comorbid concerns (e.g., depression, eat-ing disorders) that were not controlled for in theanalyses of this study.

Group leaders were provided a 1-hr trainingsession on the use of PCOMS and how to in-form clients about the research study by condi-tion. All clients referred to group therapy wereeducated about the group process by their lead-ers during a screening session, allowing them todiscuss group norms and expectations and es-tablish goals for treatment. Those interested and

found to be appropriate for the group therapyprogram were provided with instructions aboutgroup attendance and information regarding theresearch study.

Feedback condition group screening andsession protocol. Coleaders in the feedbackcondition instructed group clients on how tocomplete the ORS and GSRS and to plot theirtotal scores for these measures on a progressgraph during screenings. PCOMS graphs wereused throughout the duration of group therapyin a given semester for clients in the feedbackcondition to plot and monitor their outcomescores each session. Although these intakescores were not included in the data analysis,this process gave coleaders and clients an op-portunity to gain exposure to the measures,score, and interpret the assessments in prepara-tion for the first session.

At the beginning of each session, clientsscored and plotted their own ORS total score onthe graph to display their treatment progress.Coleaders prompted for reactions to their scores(e.g., noticed changes in their psychologicalfunctioning relative to the previous week [afterfirst session]), meaning that they had an overtconversation about their progress relative to theprevious week, particularly if below the clinicalcutoff of 25. Leaders asked group members toshare an update on their progress based on ORSTotal scores during a check in procedure as wellas to share any needs they had from the group tohelp them improve. Approximately 5 min be-fore the end of each group therapy session,clients responded to, scored, and plotted theirGSRS total scores on their graph. Again, co-leaders asked clients to share their reactions ontheir progress according to group cohesionscores relative to their previous session score(after first session) during the check-out proce-dure at the end of group.

After the third and each subsequent session(starting at Session 3 and every session there-after was evaluated relative to client Session 1ORS scores), research personnel provided a“signal system” to all group coleaders in thefeedback condition that categorized their groupmember’s progress according to manualizedprocedures (Duncan, 2011). The system is de-signed to alert coleaders to clients who wereNOT in their groups. Research personnel af-fixed color-coded index cards (described below)to client graphs with messages denoting mem-


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

bers’ progress. Member progress was deter-mined by evaluating the change from the cur-rent group session relative to the first groupsession. Coleaders were instructed to reviewsignals before each group session and discusswith one another how to incorporate the feed-back into the next session. Clinical decisionscoleaders made to assist with enhancing clients’scores were then implemented in the group ses-sion. For example, therapists with a group mem-ber who had scores indicating deterioration mayspecifically be attuned to their reported needs forthe group or overtly note concern that the memberhad not seen improvements during the process ofthe group session. Given that the nature of feed-back is designed to be individualized to thegroup member, it is not possible to capture theexact intervention implemented for each clientthat took part in the study; rather, coleaderswere trained to focus specifically on interveningwith clients who were deteriorating with thehelp of the following manualized recommenda-tions (Duncan, 2011). Each card provided asummary of how the client changed from theprevious week with recommendations for howto intervene:

No change (yellow card). Clients whohave not made at least a 5-point increase in thefirst three sessions are considered NOT and atincreased risk of treatment failure, or prematuretermination from therapy. Leaders are to ad-dress progress and inquire about clients’ per-spectives of therapy and the alliance. If clientremains unchanged after three additional ses-sions (Session 6), leaders are encouraged todiscuss the client’s perception of the group al-liance, whether the treatment format is benefi-cial, or determine whether another type of treat-ment (e.g., medication consultation) may beneeded.

Reliably deteriorated (red card). Clientswho have deteriorated by at least 5-points fromtheir baseline measure are reliably deterioratedand considered to be NOT and at risk for drop-ping out of treatment prematurely. Leaders areto address the deterioration and inquire aboutclients’ perspectives of therapy and the alliance.If clients continued to deteriorate, leaders areencouraged to speak with clients about alterna-tive treatment options.

Reliably changed (blue card). Clientswho have made at least a 5-point improvementon the ORS have experienced reliable change.

Leaders with clients in this range are to rein-force progress and specific changes clientsmade and assisted clients with further refiningtheir “change strategies.”

Clinically significant change (green card).Clients who have experienced a 5-point gainthat crossed the clinical cutoff of 25 on the ORShave made clinically significant and reliablechange, meaning clients are moving in the di-rection of “recovery.” Leaders are to assist cli-ents by refining their “change strategies” andfocus on preparing them for potential relapsesor setbacks.

Research personnel attended weekly grouptherapy supervision to promote treatment adher-ence (Hogue, Liddle, & Rowe, 1996). Specifi-cally, coleaders verbally confirmed that theydiscussed their clients’ feedback scores andprogress with one another, checked in with cli-ents about their scores on the ORS and GSRSeach session, as well as adhered to recommen-dations made via the signal system.

TAU screening and group session protocol.Group coleaders assigned to the TAU conditionprovided instructions to their clients on how tocomplete the ORS during screening appoint-ments; however, clients or coleaders neverscored the completed measures. Coleaders re-turned the completed ORS measures to researchpersonnel for scoring; they did not have accessto their clients’ progress throughout the dura-tion of the research study. Leaders not toprompt any discussions on client progress viathese measures during the study.

Statistical Analysis

Multilevel modeling methods (Heck,Thomas, & Tabitha, 2014; Hox, 2010; Rauden-bush & Bryk, 2002) were used to evaluate hy-potheses given the nested nature of the data(i.e., clients nested within groups) and potentialeffects on outcome related to data interdepen-dence (i.e., mutually dependent response patternwithin each group). Ignoring the clustered na-ture of the research design may contribute toinaccuracies in estimates of variance parameterson outcome and contribute to Type I error(Baldwin, Murray, & Shadish, 2005; Baldwin,Stice, & Rohde, 2008). A two-level model wasconstructed (clients nested within groups) todetermine if client feedback improves treatmentoutcomes in group therapy.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Three multilevel models were constructedand compared as recommended by Tasca, Il-ling, Joyce, and Ogrodniczuk (2009). First, weconstructed a baseline statistical model, or anull model (without any predictors), to estimatevariance in post ORS scores attributed to eachlevel. Next, we constructed a covariate onlymodel to determine whether variance attributedto post ORS scores was affected by the inclu-sion of a random covariate, pre ORS, or preDistress Index score. A third model was con-structed building from the covariate only model,in which the feedback variable was added as adichotomous predictor (feedback coded � 1;TAU coded � 0) at Level 2 (group level), todetermine if client feedback contributed to sta-tistically significant improved outcomes whencompared to clients in TAU.

Additional analyses were used to calculate ef-fect sizes, percentages of clients who met criteriafor reliable and clinically significant change crite-ria on the ORS, and differences in mean atten-dance scores. The calculation for effect size withingroups was: Mpost � Mpre/SDpre. The calculationused to determine the effect size of feedback, or abetween group effect size, at posttest was:B(Feedback)/�([nTAU � 1]s2

TAU post ORS �[nFeedack � 1]s2

Feedback post ORS /N � 2), the sameformula used in previous PCOMS studies evalu-ating client feedback efficacy with couples (Reeseet al., 2010). A chi-square analysis was used toevaluate differences in reliable change categoriesby condition. Also, t tests were used to comparerates of attendance and premature terminationacross feedback and TAU conditions. Missingdata were handled using a last observation car-ried forward method (Streiner, 2010), a tech-nique common in client feedback studies, inwhich the score for the last session attended isconsidered the client’s posttreatment score (e.g.,Lambert et al., 2001; Slade et al., 2008).

Results

To evaluate if randomization across groupswas successful, pre ORS scores were comparedacross conditions. No statistically significantdifferences were noted between the two treat-ment conditions for the ORS, t(82) � 0.88, p �.05, suggesting initial levels of distress wereequivalent and randomization was successful.In addition, no significant differences werenoted on client or therapist demographic vari-

ables or years of experiences in coleadershippairs across treatment conditions (p � .05).

Evaluating Efficacy

The feedback condition showed pre–post im-provement of 7.45 ORS points compared to5.24 for the TAU condition. Within conditionestimates yielded moderate to large effect sizes(Cohen, 1988) as seen in Table 1, suggestinggroup treatment was effective in both condi-tions. The between treatment condition effectsize was d � 0.41, a small-medium effect forfeedback.

Prior to evaluating the multilevel models, anintraclass correlation coefficient (ICC) was cal-culated using component estimates from thenull model to evaluate variance attributable tothe group level using the following equation:ICC � �group

2 /�client2 � �group.

2 Results indicatedthat 2% of the variance in client outcome wasattributable to differences between groups. Pa-rameter estimates from the null model foundindividual post ORS scores were significantlydifferent, � 29.1, p � .001.

Second, a covariate only model was devel-oped to evaluate if significant treatment gainswere observed on post ORS scores for the totalsample with pre ORS scores used as a covariate(grand mean centered) to control for pretreat-ment differences (see Table 2). Results suggestthat there were significant differences in postORS scores at the end of treatment ( � 29.1,p � .001), the average post ORS score was 29.0when initial functioning was controlled. A sta-tistically significant positive slope between ini-tial and final ORS scores ( � .5, p � .001)indicated that clients’ scores increased posi-

Table 1Means, Standard Deviations, and Effect Sizesfor ORS

Feedbackcondition(n� 43)

TAUcondition(n � 41)

Total(N � 84)

M SD M SD M SD

Pre ORS 23.47 7.86 22.02 7.06 22.76 7.47Post ORS 30.87 6.49 27.26 6.85 29.11 6.87Effect size (d) 0.94 0.74 0.85

Notes. TAU � treatment as usual; ORS � Outcome Rat-ing Scale; d � Cohen’s (1988) measure of sample effectsize.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

tively and significantly across treatment wheninitial level of functioning was controlled.

Third, a conditional model was constructedby adding feedback as a dichotomous predictor(feedback vs. TAU) to the covariate only modelat Level 2. The difference in post ORS scoresbetween treatment conditions was statisticallysignificant, � �2.9, p � .023. The resultsindicated that clients in the feedback conditionscored approximately 2.9 ORS points higherthan clients in the TAU group when grand meancentered pre ORS scores were controlled.

To determine if the conditional model was abetter overall fit in comparison to the covariateonly model, a chi-square statistic was used tocompare the �2LLs for the covariate onlymodel (539.38) to the conditional model(532.05). Results of this analysis, 2(1) � 7.33,p � .01, indicated that the addition of the feed-back variable when controlling for pre ORSscores improved the fit of the model.

Clinically Significant Change

We hypothesized that clients in the feedbackcondition would achieve higher rates of reliableand clinically significant change than clients inthe TAU condition. Chi-square analyses re-vealed statistically significant differences, 2(3,N � 84) � 7.6, p � .05 between clients in the

feedback and TAU conditions on classificationsbased on changes in prepost ORS scores. Spe-cifically, clients in the feedback condition onaverage achieved higher percentages of clini-cally significant and reliable change comparedto the TAU condition (see Table 3). Resultsprovide support for the second hypothesis, inthat feedback participants had higher rates ofreliable and clinically significant change on theORS.

Retention

Clients in the feedback condition were hy-pothesized to attend more sessions overall andhave lower rates of deterioration and prematuretermination compared to clients in the TAUcondition. On average, clients attended 7.33sessions (SD � 2.78; range � 2–11). Clients inthe feedback condition attended statistically sig-nificantly more sessions than clients in the TAUcondition (feedback � 8.0 and TAU � 6.6;t[82] � 2.4, p � .05).

Results indicated that feedback did not con-tribute to decreased rates of deterioration andpremature termination in this study. As can beseen in Tables 3, approximately the same num-ber of clients deteriorated in each treatmentcondition, meaning their final scores on theORS were significantly lower according to re-spective RCIs than when they began grouptreatment. The number of clients who termi-nated prematurely in the feedback (n � 7) andTAU conditions (n � 13) was not statisticallysignificant, 2(1, N � 84) � 2.7, p � .10.

Table 3Percentage of Clients Who Achieved ClinicallySignificant Change in Feedback and TAUConditions on the ORS (N � 84)

Feedbackcondition(n � 43)

TAUcondition(n � 41)

ORS classifications n % n %

Deteriorated 2 4.7 2 4.9No change 9 20.9 20 48.8Reliable change 14 32.6 7 17.1Clinically significant change 18 41.9 12 29.3

Note. ORS � Outcome Rating Scale; TAU � treatment asusual. ORS, 2(3, N � 84) � 7.6, p � .05.

Table 2Fixed and Random Effects for Two-Level MultilevelModel Predicting Postgroup Treatment ORS Scores(N � 84)

Covariate onlymodel Full model

Parameter Coefficient SE Coefficient SE

FixedIntercept (00) 29.1�� 0.8 30.6�� 0.9Pre-ORS slope (10) 0.5�� 0.1 0.4�� 0.03Feedback (01) �2.9� 1.29

Variancecomponent SD

Variancecomponent SD

RandomLevel 1 error variance

(�client2 ) 34.8 5.9 34.3 5.4

Level 2 error variance(�group) 1.4 3.1 0.0 0.0

Note. ORS � Outcome Rating Scale.� p � .05. �� p � .001.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Discussion

We had three main findings in the currentstudy. Our primary finding was that larger treat-ment gains were observed on the ORS in grouppsychotherapy for clients in the feedback con-dition compared to the TAU condition. Theseresults are consistent with the other publishedstudy that evaluated the efficacy of client feed-back using PCOMS in group psychotherapy(Schuman et al., 2014). As in Schuman et al.,statistically significant pre–post gains werefound for feedback when compared to TAU,although the magnitude of change was slightlylarger (d � 0.41 vs. d � 0.28). It is possible thatthe inclusion of the GSRS and length of groupintervention in comparison may be a contribut-ing factor to the increase in effect size; how-ever, more research to dismantle and parse outsuch effects is needed. Though the currentgroup effect size also falls in the medium range(Cohen, 1988), effects of feedback in grouptherapy appear to be more modest when com-pared to those found in RCTs using PCOMS forindividual therapy (d � 0.49 and d � 0.54;Reese et al., 2009) and couples therapy (d �0.50; Anker et al., 2009).

Study results seem to make sense when thedifferences in the intensity of the feedback in-terventions are considered in each of the stud-ies. Larger effects were observed in the individ-ual and couple studies where outcome andalliance scores were likely addressed in moredetail and as needed. Simply put, more time wasdedicated to feedback in the individual and cou-ple studies than in the current study. Con-versely, the smallest effect size was found in theSchuman et al. (2014) group study where anabbreviated feedback protocol was used andonly five sessions were evaluated. The resultsfor the current study fall somewhere in-between, although this should be interpretedcautiously given the 95% confidence intervalsof the effect sizes overlap. However, this doesseem to make intuitive sense given that thefeedback protocol was not abbreviated andmore sessions were evaluated than in Schumanet al.

The second main finding was that more cli-ents in the feedback condition experienced re-liable change (32.6% vs. 17.1%) and clinicallysignificant change (41.9% vs. 29.3%) comparedto clients in the TAU condition. The percentage

of feedback participants who achieved clini-cally significant change are almost identical toprevious study findings evaluating the efficacyof feedback using PCOMS with couples(40.8%; Reese et al., 2010). Twice as manyclients in the feedback condition achieved reli-able improvement than those in the TAU con-dition (n � 14 vs. n � 7), which is also similarto findings by Reese et al. (2009). Compared tothe Schuman et al. (2014) study, more clients inthe current study achieved reliable change(32.6% vs. 20.4%) and clinically significantchange (41.9% vs. 28.5%).

The last main finding is that feedback partic-ipants attended more group therapy sessionsthan TAU participants. Findings are similar toresults from Slade and colleagues (2008), whofound feedback clients attended 1.5 more ses-sions on average than those in the TAU condi-tion. However, we did not find significant dif-ferences between treatment conditions forpremature termination rates. Overall, the num-ber of clients who terminated prematurely (i.e.,before reaching the clinical cut-off of 25 on theORS) was low for both conditions (feedback,n � 7; TAU, n � 13). This may have been duein part to the group screening process and em-phasis placed on attending all of the sessions forthe semester.

Study Limitations

There are four major limitations in the cur-rent study. First, the biggest weakness of thisstudy was the sample size. An a priori poweranalysis conducted indicated approximately120 clients were needed for a level of power( � .80) appropriate for this study to detectan effect size of d � 0.50. The post hoc poweranalysis concluded power ( � .69) was in-sufficient to detect a medium effect size (d �0.50), therefore the results should be inter-preted with some caution. In addition, thesample was not diverse, with most of thesample being White, heterosexual identifiedwomen, thereby limiting generalizability ofresults. In addition, over half of the sample(58.3%) had already received individual ther-apy at some point prior to attending grouptherapy. It is possible that clients were in lessdistress, contributing to fewer pre–post ther-apy gains across group treatment. A smallsample size also limited our ability to conduct


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

additional analyses that would permit certaincomparisons with other feedback studies. Forexample, we attempted to evaluate whetheroutcome and retention differed for those con-sidered NOT in feedback and TAU condi-tions. However, given the small sample ofthose NOT, we were unable to conduct thiscomparison. Although the sample size issmall relative to many individual psychother-apy outcome studies, and limiting to somedegree in the current study, it is also compa-rable to those found in group therapy outcomestudies (e.g., Davies et al., 2008).

Second, is the limited generalizability ofour findings given we evaluated one type ofgroup approach (i.e., interpersonal process)provided in a university counseling center.Although it seems intuitive to conclude thatthe effects of client feedback were dilutedbecause less time was available to discuss theclient feedback measures due to the groupformat, perhaps that is not the case. Anotherpossibility is that the interpersonal processapproach to group psychotherapy, which uti-lized a strong focus on the “here and now” inboth conditions, reduced the effect of feed-back. Discussions generated using the imme-diacy technique to understand how memberswere doing in the moment perhaps mimickedor was similar to the process of overtly dis-cussing client changes on ORS and GSRSeach session. Although more research isneeded to understand this effect, the currentstudy is an improvement over the abbreviatedversion of PCOMS found in Schuman et al.(2014).

Third, treatment fidelity was not formallymonitored outside of coleaders’ verbal ac-knowledgment that they followed the studyprotocol. We attended weekly group supervi-sion to monitor the coleaders’ administrationof client feedback measures and to verballyverify that they were using the client feed-back/signal system as intended in their re-spective groups. Monitoring fidelity more for-mally through observation via videorecordings or developing a rating form toassess adherence to the protocol may haveenhanced the study design. Unfortunately,given the setting, these options were not fea-sible.

Last, there were measurement or data lim-itations of note. Although the purpose of the

study was for clients to provide feedback totheir coleaders via the PCOMS assessmentsprovided, these measures were self-reportmeasures of client functioning. Self-reportmeasures are generally known to inherentlyhave some degree of bias toward social desir-ability and lack validity in how that clientmay be objectively assessed (Barker, Pist-rang, & Elliott, 2002). In addition, a secondoutcome measure was available for use butnot included in the study, as the focus of themeasure was not consistent with the aim ofthe group therapy provided. The current re-search would be strengthened with a secondoutcome measure of interpersonal concerns.Finally, as is typical with naturalistic studies,missing data were observed between pre–posttreatment observations. We were not able tomonitor reasons for missed sessions or pre-mature termination.

Implications and Future Recommendations

Two implications from this study are noted.First, this study extends the client feedbackliterature to include results on the effects offeedback in group psychotherapy. The resultsprovide some evidence that client feedbackcan be useful for improving treatment out-comes and improving treatment retention, al-though this initial study suggests the benefitsare more modest than client feedback used inindividual or couple therapy. Second, studyfindings may have pragmatic implications toproviders governed by managed care. Usingclient feedback to enhance outcome and re-tention may well complement the efficiencyalready afforded by this widely used treat-ment modality in community resources (Tay-lor et al., 2001).

Given the findings of the current study,future research for client feedback in grouppsychotherapy is warranted. Studies withlarger and more diverse samples of clientsfrom outside of a university counseling centersetting are needed. We also suggest that amore formalized check of treatment integritybe developed to better estimate the level offidelity to the intervention as well as to facil-itate better understanding of how the processof client feedback is implemented and uti-lized. For example, observing the video re-cordings of group sessions may provide an


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

understanding of the strength of the interven-tion as well as how the intervention could bedelivered more effectively. Finally, we knowvery little about why client feedback works. Itis recommended that future studies be de-signed to isolate processes related to feedback(i.e., monitoring group cohesion) on outcome.For example, future studies should dismantleindividual constructs within the client feed-back system, such as monitoring group cohe-sion and outcome alone compared to TAU.Doing so may promote a better understandingof processes underlying client feedback, par-ticularly in the group psychotherapy format.

Conclusions

The efficacy of client feedback is substan-tial enough in individual and couple psycho-therapy that it is now considered a form ofevidence-based practice, with both the OQSystem and PCOMS being recognized by theSAMHSA National Registry of Evidence-based Programs and Practices. Client feed-back in both research and practice are consis-tent with the definition of an EBP aspurported by the APA Presidential Task Forceon Evidence-Based Practice (2006), being“the integration of best research evidencewith clinical expertise and patient values” (p.273). Although individualizing client caremay seem contradictory to the format ofgroup therapy, the use of client feedback of-fers a focus on both supporting individualneeds while also formally checking in on cli-ent perceptions of group processes. Suchpractice is well aligned with the AmericanGroup Psychotherapy Association PracticeGuidelines for Group Psychotherapy (Bernardet al., 2008; The American Group Psychother-apy Association, 2007). Specifically, Bernardet al. (2008) suggests that “in addition to thetherapist’s clinical sense, empirical assess-ment tools provide a structured approach to‘taking the pulse’ of the group interpersonalclimate to ascertain what may be obstructingor facilitating interpersonal processes at thegroup level” (p. 17). Client feedback mayserve as a method to more formally “take thepulse” of the group, to understand what isneeded both at individual and group levels topromote change in this complex format oftreatment. It is hoped that this study serves a

foundation from which additional work willbe conducted to further evaluate the efficacyof client feedback in group therapy, but alsoto understand how client feedback relates tothe processes by which change occurs in agroup format.

References

Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009).Using client feedback to improve couple therapyoutcomes: A randomized clinical trial in a natural-istic setting. Journal of Consulting and ClinicalPsychology, 77, 693–704. http://dx.doi.org/10.1037/a0016062

APA Presidential Task Force on Evidence-BasedPractice. (2006). Evidence-based practice in psy-chology. American Psychologist, 61, 271–285.http://dx.doi.org/10.1037/0003-066X.61.4.271

Baldwin, S. A., Murray, D. M., & Shadish, W. R.(2005). Empirically supported treatments or type Ierrors? Problems with the analysis of data fromgroup-administered treatments. Journal of Con-sulting and Clinical Psychology, 73, 924–935.http://dx.doi.org/10.1037/0022-006X.73.5.924

Baldwin, S. A., Stice, E., & Rohde, P. (2008). Sta-tistical analysis of group-administered interventiondata: Reanalysis of two randomized trials. Psycho-therapy Research, 18, 365–376. http://dx.doi.org/10.1080/10503300701796992

Barker, C., Pistrang, N., & Elliott, R. (2002). Re-search methods in clinical psychology: An intro-duction for students and practitioners (2nd ed.).Chichester, UK: Wiley. http://dx.doi.org/10.1002/0470013435

Bernard, H., Burlingame, G., Flores, P., Greene, L.,Joyce, A., Kobos, J. C., . . . the Science to ServiceTask Force, American Group Psychotherapy As-sociation. (2008). Clinical practice guidelines forgroup psychotherapy. International Journal ofGroup Psychotherapy, 58, 455–542. http://dx.doi.org/10.1521/ijgp.2008.58.4.455

Bordin, E. S. (1979). The generalizability of thepsychoanalytic concept of the working alliance.Psychotherapy: Research & Practice, 16, 252–260. http://dx.doi.org/10.1037/h0085885

Bringhurst, D. L., Watson, C. W., Miller, S. D., &Duncan, B. L. (2006). The reliability and valid-ity of the Outcome Rating Scale: A replicationstudy of a brief clinical measure. Journal ofBrief Therapy, 5, 23–30. Retrieved from http://journalbrieftherapy.com/

Burlingame, G. M., Strauss, B., & Joyce, A. (2013).Change mechanisms and effectiveness of smallgroup treatments. In M. J. Lambert (Ed.), Berginand Garfield’s handbook of psychotherapy and


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

behavior change (6th ed., pp. 640–689). NewYork, NY: Wiley.

Campbell, A., & Hemsley, S. (2009). Outcome Rat-ing Scale and Session Rating Scale in psycholog-ical practice: Clinical utility of ultra-brief mea-sures. Clinical Psychologist, 13, 1–9. http://dx.doi.org/10.1080/13284200802676391

Castonguay, L. G., Barkham, M., Lutz, W., & McAle-avey, A. A. (2013). Practice-oriented research: Ap-proaches and application. In M. J. Lambert (Ed.),Bergin and Garfield’s handbook of psychotherapyand behavior change (6th ed., pp. 85–133). NewYork, NY: Wiley.

Chapman, C. L., Burlingame, G. M., Gleave, R. M.,Rees, F., Beecher, M., & Porter, G. S. (2012).Clinical prediction in group psychotherapy. Psy-chotherapy Research, 22, 673–681.

Cohen, J. (1988). Statistical power analysis for thebehavioral sciences (2nd ed.). Hillsdale, NJ: Erl-baum.

Crits-Christoph, P., Ring-Kurtz, S., Hamilton, J. L.,Lambert, M. J., Gallop, R., McClure, B., . . .Rotrosen, J. (2012). A preliminary study of theeffects of individual patient-level feedback in out-patient substance abuse treatment programs. Jour-nal of Substance Abuse Treatment, 42, 301–309.http://dx.doi.org/10.1016/j.jsat.2011.09.003

Davies, D. R., Burlingame, G. M., Johnson, J. E.,Gleave, R. L., & Barlow, S. H. (2008). The effectsof a feedback intervention on group process andoutcome. Group Dynamics: Theory, Research, andPractice, 12, 141–154. http://dx.doi.org/10.1037/1089-2699.12.2.141

Duncan, B. (2011). The Partners for Change Out-come Management System (PCOMS): Administra-tion, scoring, interpreting update for the Outcomeand Session Ratings Scale. Jensen Beach, FL: Au-thor.

Duncan, B. (2012). The partners for change outcomemanagement system (PCOMS): The heart and soulof change project. Canadian Psychology/Psy-chologie canadienne, 53, 93–104. http://dx.doi.org/10.1037/a0027762

Duncan, B. (2014). On becoming a better therapist:Evidence based practice one client at a time (2ndEd.). Washington, DC: American PsychologicalAssociation.

Duncan, B. L., & Miller, S. M. (2007). The GroupSession Rating Scale. Jensen Beach, FL: Authors.

Duncan, B., & Sparks, J. (2010). Heroic clients,heroic agencies: Partners for change (2nd ed.).Jensen Beach, FL: Author.

Duncan, B., Sparks, J., Miller, S., Bohanske, R., &Claud, D. (2006). Giving youth a voice: A prelim-inary study of the reliability and validity of a briefoutcome measure for children, adolescents, andcaretakers. Journal of Brief Therapy, 5, 71–87.Retrieved from http://journalbrieftherapy.com/

Gillaspy, J. A., & Murphy, J. J. (2011). The use ofultra-brief client feedback tools in SFBT. In C. W.Franklin, T. Trepper, E. McCollum, & W. Gin-gerich (Eds.), Solution-focused brief therapy. NewYork, NY: Oxford University Press.

Hannan, C., Lambert, M. J., Harmon, C., Nielsen,S. L., Smart, D. W., Shimokawa, K., & Sutton,S. W. (2005). A lab test and algorithms for iden-tifying clients at risk for treatment failure. Journalof Clinical Psychology, 61, 155–163. http://dx.doi.org/10.1002/jclp.20108

Harmon, S. C., Lambert, M. J., Smart, D. M., Hawk-ins, E., Nielsen, S. L., Slade, K., & Lutz, W.(2007). Enhancing outcome for potential treatmentfailures: Therapist-client feedback and clinicalsupport tools. Psychotherapy Research, 17, 379–392. http://dx.doi.org/10.1080/10503300600702331

Hawkins, E. J., Lambert, M. J., Vermeersch, D. A.,Slade, K. L., & Tuttle, K. C. (2004). The thera-peutic effects of providing patient progress infor-mation to therapists and patients. PsychotherapyResearch, 14, 308–327. http://dx.doi.org/10.1093/ptr/kph027

Heck, R. H., Thomas, S. L., & Tabitha, L. N. (2014).Multilevel and longitudinal modeling with IBMSPSS (2nd ed.). New York, NY: Routledge.

Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treat-ment adherence process research in family ther-apy: A rationale and some practical guidelines.Psychotherapy: Theory, Research, Practice, 33,332–345. http://dx.doi.org/10.1037/0033-3204.33.2.332

Hox, J. (2010). Multilevel analysis: Techniques andapplications (2nd ed.). New York, NY: Routledge.

Jacobson, N. S., & Truax, P. (1991zav). Clinicalsignificance: A statistical approach to definingmeaningful change in psychotherapy research.Journal of Consulting and Clinical Psychology,59, 12–19. http://dx.doi.org/10.1037/0022-006X.59.1.12

Lambert, M. J., Hansen, N. B., & Harmon, S. C.(2010). Outcome Questionnaire System (The OQSystem): Development and practical applicationsin healthcare settings. In M. Barkham, G. E.Hardy, & J. Mellor-Clark (Eds.), Developing anddelivering practice-based evidence (pp. 141–154).Malden, MA: John Wiley & Sons, Ltd.

Lambert, M. J., Hansen, N. B., Umphress, V., Lun-nen, K., Okiishi, J., Burlingame, G. M., & Reisen-ger, C. W. (1996). Administration and scoringmanual for the OQ 45.2. Stevenson, MD: Ameri-can Professional Credentialing Services.

Lambert, M. J., & Ogles, B. M. (2004). The efficacyand effectiveness of psychotherapy. In M. J. Lam-bert (Ed.), Bergin and Garfield’s handbook of psy-chotherapy and behavior change (5th ed., pp.139–193). New York, NY: Wiley.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Lambert, M. J., & Shimokawa, K. (2011). Collectingclient feedback. Psychotherapy, 48, 72–79. http://dx.doi.org/10.1037/a0022238

Lambert, M. J., Whipple, J. L., Smart, D. W., Ver-meersch, D. A., Nielsen, S. L., & Hawkins, E. J.(2001). The effects of providing therapists withfeedback on patient progress during psychother-apy: Are outcomes enhanced? Psychotherapy Re-search, 11, 49 – 68. http://dx.doi.org/10.1080/713663852

Lambert, M. J., Whipple, J. L., Vermeersch, D. A.,Smart, D. W., Hawkins, E. J., Nielson, S. L., &Goates, M. (2002). Enhancing psychotherapy out-comes via providing feedback on client progress:A replication. Clinical Psychology & Psychother-apy, 9, 91–103. http://dx.doi.org/10.1002/cpp.324

MacKenzie, K. R. (1983). The clinical application ofa group climate measure. In R. R. Dies & K. R.MacKenzie (Eds.), Advances in group psychother-apy: Integrating research and practice (pp. 159–170). New York, NY: International Universities.

McRoberts, C., Burlingame, G. M., & Hoag, M. J.(1998). Comparative efficacy of individual andgroup psychotherapy: A meta-analytic perspective.Group Dynamics: Theory, Research, and Practice,2, 101–117. http://dx.doi.org/10.1037/1089-2699.2.2.101

Miller, S. D., & Duncan, B. L. (2000). The OutcomeRating Scale. Chicago, IL: Authors.

Miller, S. D., Duncan, B. L., Brown, K., Sparks, J., &Claud, D. (2003). The Outcome Rating Scale: Apreliminary study of the reliability, validity, andfeasibility of a brief visual analog measure. Jour-nal of Brief Therapy, 2, 91–100. Retrieved fromhttp://journalbrieftherapy.com/

Miller, S. D., Duncan, B. L., & Johnson, L. D.(2000). The Session Rating Scale 3.0. Chicago, IL:Author.

Probst, T., Lambert, M. J., Loew, T. H., Dahlbender,R. W., Göllner, R., & Tritt, K. (2013). Feedback onpatient progress and clinical support tools for ther-apists: Improved outcome for patients at risk oftreatment failure in psychosomatic in-patient ther-apy under the conditions of routine practice. Jour-nal of Psychosomatic Research, 75, 255–261.http://dx.doi.org/10.1016/j.jpsychores.2013.07.003

Raudenbush, S. W., & Bryk, A. S. (2002). Hierar-chical linear models: Applications and data anal-ysis methods (6th ed.). Newbury Park, CA: Sage.

Reese, R. J., Duncan, B., Bohanske, B., Owen, J., &Minami, T. (2014). Benchmarking outcomes in apublic behavioral health setting: Feedback as aquality improvement strategy. Journal of Consult-ing and Clinical Psychology, 82, 731–742. http://dx.doi.org/10.1037/a0036915

Reese, R. J., Norsworthy, L. A., & Rowlands, S. R.(2009). Does a continuous feedback system im-

prove psychotherapy outcome? Psychotherapy:Theory, Research, Practice, Training, 46, 418–431. http://dx.doi.org/10.1037/a0017901

Reese, R. J., Toland, M. D., Slone, N. C., & Nor-sworthy, L. A. (2010). Effect of client feedback oncouple psychotherapy outcomes. Psychotherapy:Theory, Research, Practice, Training, 47, 616–630. http://dx.doi.org/10.1037/a0021182

Schuman, D. L., Slone, N. C., Reese, R. J., & Dun-can, B. (2014). Efficacy of client feedback in grouppsychotherapy with soldiers referred for substanceabuse treatment. Psychotherapy Research. Ad-vance online publication. http://dx.doi.org/10.1080/10503307.2014.900875

Shimokawa, K., Lambert, M. J., & Smart, D. W.(2010). Enhancing treatment outcome of patientsat risk of treatment failure: Meta-analytic andmega-analytic review of psychotherapy quality as-surance system. Journal of Consulting and Clini-cal Psychology, 78, 298–311. http://dx.doi.org/10.1037/a0019247

Simon, W., Lambert, M. J., Busath, G., Vazquez, A.,Berkeljon, A., Hyer, K., . . . Berrett, M. (2013).Effects of providing patient progress feedback andclinical support tools to psychotherapists in aninpatient eating disorders treatment program: Arandomized controlled study. Psychotherapy Re-search, 23, 287–300. http://dx.doi.org/10.1080/10503307.2013.787497

Slade, K., Lambert, M. J., Harmon, S. C., Smart,D. W., & Bailey, R. (2008). Improving psycho-therapy outcome: The use of immediate electronicfeedback and revised clinical support tools. Clini-cal Psychology & Psychotherapy, 15, 287–303.http://dx.doi.org/10.1002/cpp.594

Streiner, D. L. (2010). Last observation carried for-ward. In N. J. Salkind (Ed.), Encyclopedia of re-search design (pp. 686–688). Thousand Oaks,CA: Sage.

Swift, J. K., & Greenberg, R. P. (2012). Prematurediscontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psy-chology, 80, 547–559. http://dx.doi.org/10.1037/a0028226

Tasca, G. A., Illing, V., Joyce, A. S., & Ogrodnic-zuk, J. S. (2009). Three-level multilevel growthmodels for nested change data: A guide forgroup treatment researchers. Psychotherapy Re-search, 19, 453– 461. http://dx.doi.org/10.1080/10503300902933188

Taylor, N. T., Burlingame, G. M., Fuhriman, A.,Johansen, J., & Dahl, D. (2001). A survey ofmental health care providers and managed careorganization attitudes toward, familiarity with, anduse of group interventions. International Journalof Group Therapy, 52, 243–263.

The American Group Psychotherapy Association.(2007). Practice guidelines for group psychother-


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

apy. Retrieved from http://www.agpa.org/docs/default-source/practice-resources/download-full-guidelines-(pdf-format)-group-works!-evidence-on-the-effectiveness-of-group-therapy.pdf?sfvrsn�2

Whipple, J. L., Lambert, M. J., Vermeersch, D. A.,Smart, D. W., Nielsen, S. L., & Hawkins, E. J.(2003). Improving the effects of psychotherapy:The use of early identification of treatment failureand problem-solving strategies in routine practice.Journal of Counseling Psychology, 50, 59–68.http://dx.doi.org/10.1037/0022-0167.50.1.59

Wierzbicki, M., & Pekarik, G. (1993). A meta-analysis of psychotherapy drop out. ProfessionalPsychology, Research and Practice, 24, 190–195.http://dx.doi.org/10.1037/0735-7028.24.2.190

Yalom, I. D. (1995). The theory and practice ofgroup psychotherapy (4th ed.). New York, NY:Basic Books.

Received June 24, 2014Revision received February 10, 2015

Accepted February 13, 2015 �


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Education

Slone et al. 2105