A Practical Guide to Performing Forensic Vocational Evaluations and Testifying in Court

Using Client-Focused Research Methods to Improve OutcomesSaara T. Grizzell, Ph.D., CRC, LVRC & Julie F. Smart, Ph.D., CRC (ret), LPC, LVRC, ABDA, AAPC, CCFC, NCCUtah State University

+Objectives

Specify research methods/procedures List possible uses of client-focused research ID three considerations Conceptualize approaches+OverviewIntroduce client-focused research approach and how it can be used in program evaluationResearch methods/procedures for tracking outcomes with the use of feedback (pilot study example)approaches for analyzing the data of client-focused interventions (pilot study example) what to consider when implementing successful client-focused interventions3+ Introduction

+BackgroundFirst introduced in 1996 by Howard, Moras, Brill, Martinovich, & LutzHoward et al, 1996, argued that three questions were at the root of all intervention assessments.

5+Assessing an Intervention: Three QuestionsDoes the intervention work under special, experimental conditions?

Does the intervention work in practice?

Does the intervention work for this particular consumer? (Howard, Moras, Brill, Martinovich, & Lutz, 1996)6+Setting the Context: Program EvaluationAnswers specific questions using a systematic methodTypes of QuestionsFormative and SummativeMay be conducted at several stages during a programs lifecycle

7+Applying the Three Questions to Program EvaluationDoes this program work in a special, experimental context (e.g. to meet the needs of individuals with certain types of disabilities, levels of income, certain ethnic groups, etc.)?

Does this program work in general practice?

Does the program work for this particular consumer?

8+Client-Focused Approach

+Client-Focused ApproachInvolves assessing the clients progress on a weekly basis during the treatment course and providing weekly feedback to the counselor and the client about the clients progress10+An empirically proven method for increasing the positive treatment outcomes of clients at risk for treatment failure

Treatment-focused vs. Client-focused ResearchConcerned with establishing comparative and intervention efficacy Provides aggregated results over groups of participantsFocuses on the groups response to treatment

Monitors individual progress over the course of treatment Provides feedback to the client and the clinician during the course of treatmentFocuses on client-specific response to treatment

Treatment-FocusedPatient or Client-Focused11+Client-Focused Assessment SystemsAlthough several systems have been developed, only two systems have demonstrated gains in randomized controlled trials (Duncan, 2013):The Outcome Questionnaire-45 (OQ-45; Lambert, Kahler, Harmon, Burlingame, & Shimokawa, 2013)Partners for Change Outcome Management System (PCOMS; Miller, Duncan, Sorrell, & Brown, 2005)12+Client-Focused Research: Enhancing OutcomesParticularly helpful in decreasing deterioration ratesUsed in inpatient and outpatient settingsPositively related to client outcomesUsed as a quality assurance system in agency settingsMore recently used in a state VR setting(Hawkins et al, 2004; Lambert, Hansen, & Finch, 2001a; Lambert et al, 2002a; Shimokawa, Lambert, & Smart, 2010; Whipple, Lambert, Vermeersch, Smart, Nielsen, & Hawkins, 2003; ; Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004)

13+Using Client-Focused Approach in a State VR Setting: Pilot Study Example

+Study PurposeExamine the impact of providing treatment progress feedback to 30 individuals receiving services at a vocational rehabilitation agency

15

+15Study Design and ConditionsStudy Design: Repeated measures randomized wait-list control design with matching prior to randomizationFeedback Condition (Fb): Group counseling with feedbackTreatment as Usual (TAU): Group counseling but no feedback

16+OQ-45: Progress and OutcomeOutcome Questionnaire-45 (OQ-45) designed to track progress during treatment and outcomes at termination

Responses used to generate progress report

Progress report acts as the feedback

(Lambert, Kahler, Harmon, Burlingame, & Shimokawa, 2013)17+Has Internal consistency of 0.93, a test-retest reliability of .84 (Lambert, Burlingame, Umphress, et al., 1996)

Higher scores indicate greater distress and/or frequency of symptoms or problems

(0=Never; 1=Rarely; 2=Sometimes; 3=Frequently; and 4=Almost always)

Recruitment/EnrollmentIdentified clients from own client base who met criteria Informed potential participants of studyAdministered the GRQ, a screening tool that assesses readiness for groupContacted interested participantsEnrolled participantsConducted initial interviewAdministered baseline QO-4518CounselorsResearcher+Counseling Groups5 counseling groups, 4 to 6 members each

Conducted by a facilitator and co-facilitator

Group facilitator or co-facilitator served as participants individual VR counselor

Duration and frequency: 1.5 hours per session, once weekly for 10 weeks19+Feedback ProvisionFeedback provided to counselors regarding progress of clients in Fb conditionOne page report about client treatment response

One page report provided to participants in the Fb condition A graph charting progress and a narrative about the rate of progressFeedback to CounselorsFeedback to Participants20+20End of StudyResearcher asked all participants to rate progress toward employment and group counseling goals21

+Data AnalysesThe independent variables of time, condition, and office were included in a Linear Mixed Effects Model analysis for each of the dependent variables (symptom distress, interpersonal relationships, and social role performance)22+Data AnalysesChi square analysis was used to analyze potential differences in categorical demographic variables between conditionsMcNemar test for correlated proportions was used to determine employment outcomes Independent samples t-tests were used to analyze the employment progress ratings

23+ResultsThe social role performance and mental health functioning scores in both conditions showed significant improvement.

Ratings for employment progress were statistically significant, f(251)=2.77, p=.006, two tailed

Employment outcomes were significant for both conditions (p=0.012), and close to significance for the treatment condition (p=0.063). 24+ResultsInterestingly, participants in the feedback condition who received social security or subsistence benefits made the most steady and consistent progress with interpersonal relationships (p=.025), social role performance (p=.021), and mental health functioning (p=.028).

25+25Using Client-Focused Methods in Program Evaluation

+Tasks of the EvaluatorVerify that implemented programs provide needed servicesDetermine which programs produce the most favorable outcomesSelect the programs that offer the most needed types of servicesDetermine if program interventions are the cause of the desired changesProvide information to maintain and improve quality(Posavac & Carey, 2007)+27Verify that Implemented Programs Provide Needed ServicesThe most fundamental problem with programs is that some are either never implemented as planned or are implemented in such a diluted fashion that people in need receive no or minimal benefit. (p. 4, Posavac & Carey, 2007)28+Using Client-Focused Methods to Conduct Program EvaluationsAssessing the need for the programExamining the process of meeting consumer needsAssessing what the program has actually achievedEvaluating program outcomes

29

+Considerations for Implementing a Client-Focused ApproachOn-going Monitoring: Attrition is a big concern Start with a bigger N Consider short durations of monitoringMake agreements about the logistics of routine monitoring (e.g. at office, on-line, weekly, every other week, etc.)Delivering Feedback: Consider agency and consumer resources (on-line, USPS, Log in) Implementing Feedback: Suggest a routine time for consumer and provider to meet to discuss the feedback

30+Contact InformationSaara Grizzellsaara.grizzell@aggiemail.usu.edu801-550-9786

31

+