Webinar June 25, 2014 Cricket Mitchell, PhD Senior Associate, CiMH

Webinar June 25, 2014

Cricket Mitchell, PhD

Senior Associate, CiMH

Understanding the CiMH Dashboard Reports for LA PEI MAP

Some technical issues…• All participants are in a muted, “listen only” mode• Questions can be submitted at any time using the

function on your control panel– I will do my best to address all questions

• You can also ‘Raise your Hand’ via the control panel– At the end of the webinar, I will attempt to un-mute

those with their hands raised, to address questions verbally and/or allow for discussion

• Today’s webinar is being recorded and will be posted on CiMH’s website and YouTube channel

2

Purpose of Today’s Webinar• The CiMH Program Performance and Outcome Evaluation

Dashboard Reports for MAP can be confusing• We want to ensure that everyone interested in data from

the LA PEI MAP implementation can make sense of the reports– This includes county administrators, provider administrators,

program managers, MAP supervisors, MAP therapists, data entry personnel, and PracticeWise… just to name a few…

• Anyone who is not able to participate in today’s live webinar, can download the materials and review the information in their own timeWe’re going to cover a lot of EXCITING EXCITING material today… so let’s get started!!

3

CiMH Dashboard Reports• Burning questions in the minds of MAP

practitioners, supervisors, and administrators across the County….– What the heck do all those tables mean?– So… how are we doin’?– I’ve got my site report… how can I use it?

4

Understanding the CiMH MAP Dashboard Reports• All attendees registered for today’s webinar

received this PowerPoint presentation and the most recent Aggregate MAP report (also the final CiMH-generated MAP report)– Data thru December 2013

• During today’s webinar we will go back and forth between the PowerPoint presentation and the most recent Aggregate report

5

Understanding the CiMH MAP Dashboard Reports• Aggregate Report, Page 1

– All agencies that have submitted data are listed – Some agencies submit multiple data shells

reflecting MAP services at different sites; and, some receive more than one site report (upon request)

– Some agencies submit more data than requested (i.e., clients with service dates that are later than the date specified by the submission interval)

• All data received are included in the report (“at minimum, clients served through…” )

6

Understanding the CiMH MAP Dashboard Reports• Table 1: Entry and Dropout Rates

– Overall MAP Entry Rate is defined as clients who were referred to MAP and have a first session documented in one of the four treatment targets approved under LA PEI

• No Date of First Session, client did not enter MAP• No treatment target approved under LA PEI, client did

not enter MAP– Because the data shells do not track the

treatment target to which a client is referred, we are not able to determine the Entry Rate for each treatment target

7

Understanding the CiMH MAP Dashboard Reports• Table 1

– Overall MAP Dropout Rate is defined as clients who entered MAP but did not complete the full intervention as determined by the therapist

• In the data shell: “No” for “Completed MAP” without an indication that the client has Begun Another Treatment Focus

– Each target-specific Dropout Rate is defined as clients who have a first session documented in the treatment focus, but did not complete the full intervention targeted on that treatment focus as determined by the therapist

• In the data shell: “No” for “Completed Focus #”

8

Understanding the CiMH MAP Dashboard Reports• Table 1 (continued)

– The number of clients who dropped out each treatment focus does not equal the number of clients who dropped out of MAP

• It is possible to drop out of a treatment focus and not drop out of MAP

• Some clients are identified in the data shell as dropping out of one or more treatment focus, yet are also identified as successfully completing MAP

• Some clients have dropped out of one or more treatment focus, and are still receiving services under the MAP model

9

Understanding the CiMH MAP Dashboard Reports• Table 2: Treatment Focus Distribution

– It was expected that all clients who entered MAP would have service delivery information for Focus #1

– The number of clients who entered MAP should match the number of clients in the Focus #1 column

– At the most recent data submission, some clients had service delivery information at Focus #2 without data entered for Focus #1

• This explains the difference between 19,410 clients that entered MAP (in Table 1) and 19,407 clients with a Focus #1 in Table 2

10


– The vast majority of MAP clients have been or are being served for only one treatment target/focus

• 4.9% have a Focus #2, and less than 1% have a Focus #3 or Focus #4

– Most MAP clients are served for Disruptive Behaviors

• 43.0% of those with a valid Focus #1– The smallest group of MAP clients are served for

Trauma• 4.0% of those with a Focus #1

11

Understanding the CiMH MAP Dashboard Reports• Table 3: Client Demographics

– The title of the table indicates that this information is reported for clients who Entered MAP

– The left-hand column includes the number of clients (N) that Entered Overall MAP, and the number of clients that entered each Treatment Target

• The number of clients who entered all four treatment targets does not equal the number of clients who entered MAP

– It is possible for a client to be served across more than one treatment target approved under LA PEI

– Some clients are indicated as having entered the same treatment target at more than one Focus #

12


– We don’t have demographics on all MAP clients• Age is calculated as the difference between the Date of First

Session and the client’s Date of Birth– If one or both of these dates are missing or incorrect, we cannot

calculate age– This explains the lower n for age than the N for Entered (e.g., we

can only calculate age for 19,118 clients, but we have data to indicate that 19,410 clients Entered Overall MAP)

• The percentages across Gender (Female and Male) may not total 100 due to missing data

• The percentages across Ethnicity (African American, Asian/ Pacific Islander, Caucasian, Hispanic/Latino and Other) may not total 100 due to missing data

13

Understanding the CiMH MAP Dashboard Reports• Table 4: DSM-IV Primary Axis I Diagnoses

– As with Table 3, the title indicates that these data are reported for clients who Entered MAP

– The N’s for each group are reported in the far left-hand column, as they were in Table 3

– Because there is a “Missing” category, the percentages across each row of diagnostic categories will total 100

14

Understanding the CiMH MAP Dashboard Reports

• Table 5: Process Data– The title of the table indicates that these data are reported for

clients who Entered MAP; but, the denominator is reduced to only include clients who were served for that treatment focus and are within the valid age range for the measure(s)

– Each column includes parent and/or client self-report; in other words, agencies are credited for collecting pre- data from only one informant, even if both are required

– Collecting pre- data on the required outcomemeasures is an important aspect of the MAP process

– MAP treatment outcomes cannot be reported if pre-data are not collected

15


– The number of clients who have a valid pre- outcome measure must meet the following criteria:

• The client must be within the valid age range for the measure• The score must be valid (i.e., must be within the possible

range of scores for the measure)• The measure must match the treatment target for which it is

required under LA PEI (e.g., an RCADS is only included for clients treated for Anxiety)

– Appendix A includes descriptions of all required outcome measures for MAP under LA PEI

16


• Table 6: Service Delivery Data– The title of the table indicates that these data are reported

for clients who Completed; however, the number of clients (N) who Entered each group is again reported in the far left-hand column (as in Tables 3 and 4)

– The Number of Clients who Completed is reported in the first data column

• Almost 8,000 MAP clients are currently in treatment– 19,410 clients Entered MAP; 5,621 have Completed MAP;

and, 5,866 have Dropped Out (as reported in Table 1)– This indicates that there were 7,923 open cases at the

time of the data submission

17


– Completed Overall MAP is defined as “Yes” for “Completed MAP” at any of the four treatment foci service delivery sections in the data shell

– Completed for each Treatment Target is defined as “Yes” for “Completed Focus #” at any of the four treatment foci service delivery sections in the data shell

18


– Length of Treatment is calculated as the difference between the Date of Last Session and Date of First Session within each Treatment Focus; and, for Overall MAP it is summed across treatment foci.

• If one or both of these dates are missing or incorrect, we cannot calculate Length of Treatment

– Number of Sessions is a field in the data shell• If it is missing, we cannot report number of sessions

19


– The data reported in this table don’t make sense!• The range for number of weeks in treatment is as low

as 1• The range for number of sessions is as low as 1• BUT… only clients who are indicated as having

Completed are included…??????• In addition, the upper limits for the range of number of

weeks in treatment (300) and number of sessions (339) are higherhigher than what we would expect for model-adherent service delivery under MAP

20

Understanding the CiMH MAP Dashboard Reports• Tables 7-11: Outcome Data

– The table titles indicate that outcomes are reported for clients who have Completed (Overall MAP or Target-Specific Treatment); and, the number of clients who have Completed treatment within each group is reported in parentheses at the end of the title (these n’s match the n’s for Completed in Table 6)

– Three Outcome Indicators are reported in the outcome data tables

• Appendix A describes these Outcome Indicators– The minimum n necessary to report outcomes in the

aggregate report is 20• This is decreased to 5 for site reports… Even though outcomes

based on a small n are unreliable, we want to give sites as much of their data back in their site reports as possible

21

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Percent Improvement

– The percent change from the average pre- score to the average post- score is reported for each outcome measure

– Change is always reported in terms of improvement– Interpreting percent change can be difficult

• It is influenced by the range of scores – a measure with a smaller range of scores is likely to show a greater percent change than a measure with a larger range of scores

• It does not translate to clinically meaningful change– It is an overall reflection of treatment success

• Do our clients get better after participation in this treatment?

22

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Percent Improvement (cont’d)

– Two additional pieces of information are provided under each Percent Improvement: the (n), or number of clients, included in the analysis; and, the average [pre- score] on the measure

• Only clients who have valid pre- and post- data, and are indicated as having Completed treatment are included

• Knowing the average pre- score…– Indicates whether or not clients were, on average, above or

below the clinical cutpoint prior to treatment– Allows you to calculate the post- score

23

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Percent Improvement (cont’d)

– Some percentages have an asterisk (*) next to them– A paired t-test analysis is conducted with each set of

pre/post scores; and, when the difference observed is not likely to be due to chance (p<.01), this is indicated with an asterisk and a footnote

– In addition to reporting Percent Improvement in the Outcome Data Tables, there is a Graph that represents each pre/post difference

• The average pre- score and the average post- score are presented in bar graphs for each measure

• If the measure has a clinical cutpoint, it is indicated with a solid horizontal line

24

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Effect Size Estimate

– Cohen’s d a standardized effect size measure that estimates the magnitude, or strength, of a relationship

• A nice addition to reporting Percent Improvement, which is not a standardized measure

– For MAP, it is an estimate of the strength of the relationship between the average pre- score and the average post- score, expressed in terms of standard deviations

• An effect size of .5 indicates that the average pre- score is .5 standard deviations greater than the average post-score

– Think back to those stats classes!

25

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Effect Size Estimate (cont’d)

– While there is no absolute agreement about what magnitude of an effect size is necessary to establish clinical significance, conventional interpretations of Cohen’s d are

• .2 - .3 represents a small effect• .5 reflects a medium effect• .8 or greater represents a large effect

– The same clients included in Percent Improvement are included in the Effect Size Estimate

26

Understanding the CiMH MAP Dashboard Reports• Outcome Indicator: Reliable Change

– We can look at two scores and see that they are different; but, how do we know that the difference we’re observing isn’t simply due to measurement error? How much difference do we need to observe to know that it is a reliable amount of change?

– A fairly complex formula is used that takes into account the variability of the pre- treatment group and measurement error, resulting in our ability to categorize pre/post change for each measure into three groups:

• Reliable Positive Change, Reliable Negative Change, and No Reliable Change

27


• Outcome Indicator: Reliable Change (cont’d)– The same clients included in Percent Improvement and

Effect Size Estimate are included in Reliable Change– In addition to reporting the percent of clients with Reliable

Change in each of the three groups, there is a Graph to represent each set of Reliable Change data

• However, these data are presented differently than the other outcomes

• Reliable Change groups are graphed in 6-month intervals– Date of Last Session is used to determine the interval in which a client is

included– The n’s across all 6-month intervals may not equal the n in the outcomes

table» All data are included in the outcomes tables, even for clients with

service delivery dates that are later than the minimum date required at the data submission interval

28

So… How are We Doing?• We have enough data to report outcomes

for all treatment targets!

• The data indicate, on average, clients improve after participation in treatment using the MAP model– They improve in overall mental health

functioning, and in target-specific areas, such as anxiety, depression, disruptive behavior, and post-traumatic stress symptoms

29

So… How are We Doing?• On average, clients are entering MAP treatment with

scores in the clinical range on the measure of overall mental health functioning; and, completing MAP with scores below the clinical cutpoint– According to both parent report and

youth self-report

• The target-specific outcomes are a bit different, with clients starting out below the clinical cutpoint on most measures; and, staying below the clinical cutpoint after MAP (on average)– The exception is Disruptive Behavior, with clients starting

out right at the clinical cutpoint prior to MAP, and dropping below the cutpoint after treatment

30

So… How are We Doing?• Looking at the Reliable Change data…• The majority of clients with Overall MAP outcomes

have an amount of change on the YOQs from pre- to post- treatment that is Positive– 58% Positive Change according to parent report– 53% Positive Change according to youth self-reports

• The positive change groups are not as large for the target-specific outcome measures; however, clients are not (on average) beginning treatment in the clinical range on these, as they are on the YOQs– Again, the exception is for Disruptive Behavior, with

53-56% of parents reporting Positive Change

31

So… How are We Doing?

• LA PEI MAP outcomes are looking good…

• However, the number of clients for whom we have outcome data is less than half of those who have Completed MAP

Entered MAPN=19,410

Completed MAPn=5,621

YOQ Outcome Datan=2,724

(29% of clientswho Entered MAP)

(48% of clientswho Completed MAP)

32

How Can I Use My Site Report?• The MAP reports differ from other CiMH

dashboards…– The Aggregate data are provided along with your

site-specific data, so you can readily compare how your program is performing relative to the average MAP program in the County

• We try to fit both sets of data on the same page for ease of viewing; however, sometimes that’s just not possible!

– Prompts throughout the report provide guidance on interpreting the data within the context of the MAP model

33

34

Interpretation in the MAP ContextEntry and Dropout rates correspond to the MAP decisions about engagement and crisis. The MAP suggests an increased focus on those youth who drop out to identify strategies to keep them engaged in services. Further, if entry rates are below expected values (are we underutilizing MAP?), similar recruitment or engagement strategies should be considered.

Interpretation in the MAP ContextThe Measure Completion by Treatment Focus distribution corresponds to the MAP decisions about the availability and use of client information. The MAP emphasizes client assessment as “the first evidence-based practice” to generate information for focused treatment planning and monitoring. The MAP suggests an increased focus on measurement and assessment if client information is limited or missing.

ClinicalProgress?

Service Delivery,Outcome Data

Continue plan until goals met

Disengaged?Crisis?

Pursue engagement,Take appropriate action

Focus by Diagnosis

Poor Treatment

Fit?

TreatmentIntegrity?

Agency Info Re: MAP(PPMT, PG)

Add consultation or training supports

Identify barriers and revise plan

Increase supports, Change intervention, Further consultation,

Add intervention

Plan Unfocused?

Identify targets,Set goals,

Select interventions

TreatmentFocus

Client InfoNeeded?

Select measures, Perform assessment

Demographics, Measure Complete

Entry & Dropout,Service Delivery

Interpretation in the MAP ContextService Delivery data correspond to the MAP decisions about engagement and clinical progress. For example, unexpectedly low service use (e.g., very short treatment length, very few sessions) could be indications of limited engagement, obstacles related to access or availability, or insufficient service intensity. If Overall MAP treatment length exceeds the lengths for each specific treatment focus, multiple treatment foci may be the norm, and investigation of problem identification (including comorbidity), target selection, and focus management may be warranted. In terms of progress, long lengths of treatment may indicate limited client progress and slow rate of change. Otherwise, if service delivery data are consistent with expectations, the MAP emphasizes continued pursuit of current plans and goals.

Examples of prompts throughout site reports that guide interpretation of the

data within the context of the MAP model

MAP Process Guide

How Can I Use My Site Report?

• Each site report in this final set is comprehensive…– All outcome data tables and graphs are

included, even if a site did not have data for that treatment focus and/or measure

– This ensures that each site report contains all Aggregate LA PEI MAP data

35

How Can I Use My Site Report?

• Do your data look accurate?– The first level of program performance

reflected in these reports is the extent to which data are being collected and entered accurately and completely

• There might be cause to go back and look at your data shell

• If something doesn’t look like its accurately reported, schedule a call with Cricket

– Even though there will be no more submissions to CiMH, we are happy to help investigate potential errors in data entry that can be corrected in the PEI OMA

36

How Can I Use My Site Report?• Compare your site-specific data with the

aggregate data– Is your program performing on par with the average

MAP program in the County?• There may be unique factors within an agency or site’s

implementation that provide context for interpreting the data

– Are you achieving positive outcomes on par with the average MAP program in the County?

• There are still many sites who do not have enough outcome data to report; and, many who only have outcomes for a very small number of clients – BE CAUTIOUS IN INTERPRETING DATA BASED ON SMALL NUMBERS OF CLIENTS

37

How Can I Use My Site Report?• Is your entry rate low? (Table 1)• Is your dropout rate high? (Table 1)• Is the distribution of demographic

characteristics reflective of clients in need of services (within your agency and/or Service Area)? (Table 3)

• What is the distribution of Primary Axis I diagnoses relative to your treatment focus distribution? (Tables 4 and 2)

• Are you collecting pre- data on the required outcome measures? (Table 5)

38

How Can I Use My Site Report?• For clients who have Completed MAP

– Does your service delivery data look appropriate? (Table 6)

– What proportion have outcome data? (Table 7 - divide the n for YOQs by the number who have completed; in the most recent aggregate report, this is 2,724 ÷ 5,623)

• For clients who have Completed a Treatment Target– Does your service delivery data look appropriate?

(Table 6)– What proportion have outcome data?

(Tables 8-11, same process as for Table 7 – the n for completed is in the table header, and the n for an outcome measure is in parentheses below the percent improvement)

39

What if I Still Have Questions?

• Don’t hesitate to contact Cricket– The MAP reports are dense and can be

confusing (hopefully a little less so after today)

– Ask questions about the data in your site report…

– Ask questions about the outcome measures…

– Ask for help finding an error in your data entry…

– We’re here to help!!

40

41

The EndThe End

Contact Information•Cricket Mitchell, PhD

•Email: [email protected]•Cell phone: 858-220-6355

mailto:[email protected]

Documents

Webinar June 25, 2014 Cricket Mitchell, PhD Senior Associate, CiMH