19th Annual RTC Conference Presented in Tampa, …rtckids.fmhi.usf.edu/rtcconference/handouts/pdf/19...19th Annual RTC Conference Presented in Tampa, February 2006 1 CASRC Multiple

19th Annual RTC Conference

Presented in Tampa, February 2006

1

CASRC Multiple Stakeholder PerspectivesMultiple Stakeholder Perspectives

on Evidence-Based Practiceon Evidence-Based Practice

ImplementationImplementation

Gregory A. Aarons, Ph.D.Gregory A. Aarons, Ph.D.1,2,31,2,3

Karen Karen ZagurskyZagursky, B.A. , B.A. 1,31,3

Larry Palinkas, Ph.D. Larry Palinkas, Ph.D. 1,41,4

11Child and Adolescent Services Research Center (CASRC)Child and Adolescent Services Research Center (CASRC)11ChildrenChildren’’s Hospital San Diego, CAs Hospital San Diego, CA

22San Diego State UniversitySan Diego State University33University of California, San DiegoUniversity of California, San Diego

44University of Southern CaliforniaUniversity of Southern California1,1,

3020 Children’s Way, MC 5033

San Diego, CA 92123

[email protected]

(858) 966-7703 ext. 3550

http://casrc.org

CASRC

AcknowledgementsAcknowledgements

NIMH NIMH –– R03 R03 MH070703MH070703 (Aarons) Concept (Aarons) ConceptMapping of Readiness for Evidence-BasedMapping of Readiness for Evidence-BasedPracticePractice

NIMH - R01 MH072961 (Aarons) Mixed-NIMH - R01 MH072961 (Aarons) Mixed-Methods Study of a Statewide EBPMethods Study of a Statewide EBPImplementationImplementation

NIMH P30 MH068579 (Proctor) Pilot Study:NIMH P30 MH068579 (Proctor) Pilot Study:Aarons PI Organizational Receptivity toAarons PI Organizational Receptivity toEvidence-Based PracticeEvidence-Based Practice

CASRC

AgendaAgenda

The need for effective implementationThe need for effective implementation

Barriers and Facilitators to Implementing EBPBarriers and Facilitators to Implementing EBP

Study methodsStudy methods

ResultsResults

What does it all mean?What does it all mean?

CASRC

EBP Implementation is HappeningEBP Implementation is Happening

Effective implementation of EBPs into real-world serviceEffective implementation of EBPs into real-world servicesettings is important for improving service quality andsettings is important for improving service quality andoutcomes for youth outcomes for youth (Hoagwood & Olin, 2002; Jensen, 2003)(Hoagwood & Olin, 2002; Jensen, 2003)

Some (but not many) implementation improvementSome (but not many) implementation improvementmethods are being tried methods are being tried (Haynes & Haines, 1998)(Haynes & Haines, 1998)

–– Abstracting servicesAbstracting services

–– Evidence-based clinical guidelinesEvidence-based clinical guidelines

–– Incentives for better care systemsIncentives for better care systems

–– Increasing effectiveness of quality improvement programsIncreasing effectiveness of quality improvement programs

Research is testing some factors associated withResearch is testing some factors associated withimplementation but multiple stakeholder perspectives areimplementation but multiple stakeholder perspectives arenot well defined not well defined (NIMH R01, R03, PI: Aarons; R01 Webster Stratton,(NIMH R01, R03, PI: Aarons; R01 Webster Stratton,R01 Chaffin, R34 Shipp, )R01 Chaffin, R34 Shipp, )

CASRC We are Learning about ImplementationWe are Learning about Implementation

Some barriers to implementation have beenSome barriers to implementation have beenidentifiedidentified

–– e.g., lack of funds for continuing education (Simpson,e.g., lack of funds for continuing education (Simpson,2002).2002).

We know little about the most effective mannerWe know little about the most effective mannerin which to implement EBPsin which to implement EBPs

–– ((HenggelerHenggeler, Lee, & Burns, 2002; Morgenstern, 2000), Lee, & Burns, 2002; Morgenstern, 2000)

New models of implementation have beenNew models of implementation have beendevelopeddeveloped

–– (Aarons, 2005; (Aarons, 2005; FrambachFrambach & & SchillewaertSchillewaert, 2002; Klein,, 2002; Klein,ConnConn,& ,& SorraSorra, 2002)., 2002).

CASRC

Implementation is ComplexImplementation is Complex

Implementation should be evidence-basedImplementation should be evidence-based

Implementation is a multilevel issue (Dixon et al., 2001).Implementation is a multilevel issue (Dixon et al., 2001).–– PoliciesPolicies

–– AgenciesAgencies

–– ProgramsPrograms

–– Administrative staffAdministrative staff

–– CliniciansClinicians

–– ConsumersConsumers

Clear, comprehensive, measurable, and testableClear, comprehensive, measurable, and testableimplementation models are needed to guide research onimplementation models are needed to guide research onorganizational changeorganizational change

There are few empirical studies addressing these issues inThere are few empirical studies addressing these issues inyouth mental health servicesyouth mental health services



2

CASRC

Goals of the StudyGoals of the Study

To identify barriers and facilitators ofTo identify barriers and facilitators of

adoption of EBPs for organizations servingadoption of EBPs for organizations serving

youth with Mental Health disordersyouth with Mental Health disorders

Examine what various stakeholder groupsExamine what various stakeholder groups

identify as most important and mostidentify as most important and most

changeable.changeable.

CASRC

Methods IMethods I

Programs within agencies selected based on:Programs within agencies selected based on:

–– Types of Services ProvidedTypes of Services ProvidedOutpatientOutpatient

Day TreatmentDay Treatment

Case ManagementCase Management

Residential/InpatientResidential/Inpatient

–– Size of AgencySize of AgencyLarge and SmallLarge and Small

–– Size of ProgramSize of ProgramLarge and SmallLarge and Small

–– LocationLocationUrban vs. RuralUrban vs. Rural

CASRC

Participant SelectionParticipant Selection

Selected programs were either operated by theSelected programs were either operated by theCounty or provided contract services to theCounty or provided contract services to thecounty.county.

Organizational structures varied by level ofOrganizational structures varied by level ofbureaucracy and fiscal constraints on servicesbureaucracy and fiscal constraints on services(Aarons, 2004)(Aarons, 2004)

Individual participants selected by snowballIndividual participants selected by snowballsamplingsampling

CASRC

Sample SelectionSample Selection

Participants drawn from 6 organizationalParticipants drawn from 6 organizationallevels:levels:

Policy: County Mental Health Officials (n = 6)Policy: County Mental Health Officials (n = 6)

Agency: Organization/Agency directors (n = 5)Agency: Organization/Agency directors (n = 5)

Program: Program managers (n = 6)Program: Program managers (n = 6)

Clinical: Clinicians (n = 7)Clinical: Clinicians (n = 7)

Administrative: Administrative staff (n = 3)Administrative: Administrative staff (n = 3)

Consumers: Consumers of MH services (n = 5)Consumers: Consumers of MH services (n = 5)

CASRC

9.73Other

3.21Asian American

3.21African American

9.73Hispanic

74.223Caucasian

Race

27-6010.944.4Age

61.319Female

38.712Male

Gender

RangeSDMean%N

Demographics (N=31)Demographics (N=31)CASRC

7.72To a great extent

30.88To a moderate extent

30.88To a slight extent

30.88Not at all

Experience Implementing EBPs

RangeSDMean%N

Demographics Mental HealthDemographics Mental Health



3

CASRC

ProcedureProcedure

Concept Mapping Concept Mapping ((TrochimTrochim, Cook, & , Cook, & SetzeSetze, 1994), 1994)

–– Mixed qualitative-quantitative methodMixed qualitative-quantitative method

–– Qualitative methods used to generate dataQualitative methods used to generate data

–– Data analyzed using quantitative methodsData analyzed using quantitative methods

Begin with structured brainstormingBegin with structured brainstorming

–– Participants Participants generategenerate and then and then useuse a focus a focus

statement to guide identifying barriers andstatement to guide identifying barriers and

facilitators to implementationfacilitators to implementation

CASRC

ProcedureProcedure

Focus statementFocus statement

–– ““What are the factors that influence theWhat are the factors that influence the

acceptance and use of evidence-basedacceptance and use of evidence-based

practices in publicly funded mental healthpractices in publicly funded mental health

programs for families and children?programs for families and children?””

Independent stakeholder groupIndependent stakeholder group

brainstormingbrainstorming

Statements combined across all groupsStatements combined across all groups

CASRC

ProcedureProcedure

““UnstructuredUnstructured”” Card Sort Card Sort

–– 105 Statements105 Statements

–– All participants sort the same statementsAll participants sort the same statements

–– Sorted based on similaritySorted based on similarity

–– >1 pile>1 pile

Statement RatingsStatement Ratings

–– "Importance"Importance““

–– "Changeability""Changeability"

–– 0 to 4 point scale0 to 4 point scale(Not at all (Not at all !! A very great extent) A very great extent)

CASRC AnalysisAnalysis

Multidimensional scaling (MDS) and cluster analysisMultidimensional scaling (MDS) and cluster analysis

MDS analysis results in a MDS analysis results in a ““mapmap”” of the conceptual space of the conceptual spacewith similar issues closer togetherwith similar issues closer together

Solution represents psychological Solution represents psychological ““distancedistance”” or similarity or similaritybetween conceptsbetween concepts

Statements more similar in meaning are closer togetherStatements more similar in meaning are closer together

Statements grouped into non-overlapping categoriesStatements grouped into non-overlapping categoriescalled clusterscalled clusters

Clusters closer together are more conceptually relatedClusters closer together are more conceptually related

CASRC

ResultsResults

Fourteen overall clusters best fit dataFourteen overall clusters best fit data

One overall solution for all participantsOne overall solution for all participants

–– Participants reconvene to Participants reconvene to ““make sensemake sense”” of of

solutionsolution

–– Cluster namingCluster naming

Importance ratings overlaid on solutionImportance ratings overlaid on solution

CASRC

14 Clusters14 Clusters

"#"# Clinical PerceptionsClinical Perceptions

$#$# Staff Development & SupportStaff Development & Support

%#%# Staffing ResourcesStaffing Resources

&#&# Agency CompatibilityAgency Compatibility

'#'# EBP LimitationsEBP Limitations

(#(# Consumer ConcernsConsumer Concerns

)#)# Impact on Clinical PracticeImpact on Clinical Practice

*#*# Beneficial Features (of EBP)Beneficial Features (of EBP)

+#+# Consumer Values & MarketingConsumer Values & Marketing

",#",#System Readiness & CompatibilitySystem Readiness & Compatibility

""#""#Research & OutcomesResearch & Outcomes

"$#"$#Political DynamicsPolitical Dynamics

"%#"%#FundingFunding

"&#"&#Costs of EBPCosts of EBP



4

CASRC

1

2

3

4

5

6

7

8

9

10

11

12

13

14

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27 28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

Point and Cluster Map CASRC

1 Clinical Perceptions

2 Staff Development & Support

3 Staffing Resources

4 Agency Compatibility

5 EBP Limitations

6 Consumer Concerns

7 Impact on Clinical Practice8 Beneficial features (of EBP)

9 Consumer Values & Marketing

10 System Readiness & Compatibility

11 Research & Outcomes Supporting EBP

12 Political Dynamics

13 Funding14 Costs of EBP

Cluster Legend Layer Value

1 2.68 to 2.78

2 2.78 to 2.87

3 2.87 to 2.97

4 2.97 to 3.07 5 3.07 to 3.17

Figure 1: Overall Solution with Figure 1: Overall Solution with Importance Ratings

CASRC

Consumer Values & Marketing

Research & Outcomes Supporting EBP

Consumer Concerns

Impact on Clinical Practice

Clinical Perceptions

Staff Development & Support

Staffing Resources

Agency Compatibility

Costs of EBP

Funding

System Readiness

Compatibility

Beneficial Features (of EBP)

Political Dynamics

Figure 2: County Officials

EBP Limitations

CASRC



Consumer Concerns

Impact on Clinical PracticeBeneficial Features (of EBP)

Political Dynamics

Funding

Costs of EBP


EBP Limitations


Staffing Resources


System Readiness &

Compatibility

Figure 3: Agency Directors

CASRC Figure 4: Program Managers



Consumer Concerns



Staffing Resources


EBP Limitations


Political Dynamics

Funding

Costs of EBP


System Readiness &

Compatibility

CASRC Figure 5: Clinicians



Consumer Concerns


EBP Limitations




Staffing Resources

System Readiness & Compatibility

Political Dynamics

Funding

Costs of EBP




5

CASRCFigure 6: Administrative Staff



Consumer Concerns




Staffing Resources

EBP Limitations


Costs of EBP

Funding

Political Dynamics



CASRC Figure 7: Consumers

Funding

Costs of EBP


Staffing Resources




EBP Limitations

Beneficial Features

(of EBP)



Political Dynamics

Consumer Concerns


CASRC Importance Rating ScaleImportance Rating Scale

3.103.062.812.603.002.502.81System Readiness &

Compatibility

3.083.273.053.293.263.063.16Staffing Resources

3.323.473.153.093.122.963.16Staff Development & Support

3.223.152.953.213.112.913.09Research & Outcomes

Supporting EBP

2.803.222.782.953.132.672.90Political Dynamics

3.383.333.022.592.802.002.81Impact on Clinical Practice

3.333.712.943.003.253.133.17Funding

2.533.112.722.672.802.532.70EBP Limitations

3.093.562.963.083.422.913.13Costs of EBP

3.473.112.672.812.732.602.87Consumer Values & Marketing

3.033.262.872.842.672.592.85Consumer Concerns

2.953.333.082.822.532.702.88Clinical Perceptions

3.403.112.783.052.532.802.94Beneficial features (of EBP)

2.913.112.722.542.642.362.68Agency Compatibility

CnsmrAdminClincnPrgm

Mgr

Agncy

Dir

Cnty

Offcls

All

CASRC

ResultsResults

For the overall group, Funding was rated theFor the overall group, Funding was rated the

most important factor and rated the leastmost important factor and rated the least

changeable.changeable.

Staffing Resources and Staff Development andStaffing Resources and Staff Development and

Support were rated most important after funding.Support were rated most important after funding.

Clinical Perceptions and Consumer Values andClinical Perceptions and Consumer Values and

Marketing were rated most changeableMarketing were rated most changeable

Staff Development and Support ranked third inStaff Development and Support ranked third in

importance and fourth in changeabilityimportance and fourth in changeability

CASRC

ConclusionConclusion

Found a common solution that representsFound a common solution that represents

multiple stakeholder perspectivesmultiple stakeholder perspectives

There are a number of multiple stakeholderThere are a number of multiple stakeholder

concerns that may impact implementation ofconcerns that may impact implementation of

EBPs in real world service settings.EBPs in real world service settings.

Groups varied on Importance and ChangeabilityGroups varied on Importance and Changeability

ratings.ratings.

It is important to consider the concerns ofIt is important to consider the concerns of

multiple stakeholders in EBP implementation.multiple stakeholders in EBP implementation.

CASRC

ConclusionsConclusions

Processes for egalitarian multiple stakeholders input canProcesses for egalitarian multiple stakeholders input canfacilitate cultural exchangefacilitate cultural exchange

Stakeholder perspectives can inform implementationStakeholder perspectives can inform implementationprocessprocess

Examples:Examples:

–– Optimizing message content may promote more positiveOptimizing message content may promote more positiveattitudes toward implementation of change in service modelsattitudes toward implementation of change in service models

–– Staff issues need to be addressed up front to promoteStaff issues need to be addressed up front to promoteimplementation effectivenessimplementation effectiveness

Further research is needed to better understand howFurther research is needed to better understand howfactors identified in the present study impact actual EBPfactors identified in the present study impact actual EBPimplementation efforts.implementation efforts.



6

CASRC Reference List

Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Organizational

context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14,

255-271.

Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice:

The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61-

74.

Dixon, L., McFarlane, W. R., Lefley, H., Lucksted, A., Cohen, M., Falloon, I. et al. (2001). Evidence-

based practices for services to families of people with psychiatric disabilities. Psychiatric Services,

52(7), 903-910.

Haynes, B., & Haines, A. (1998). Barriers and bridges to evidence based clinical practice. Biomedical

Journal, 317, 273-276.

Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after the innovation is identified?

Clinical Psychology: Science and Practice, 9(2), 191-194.

Hoagwood, K., & Olin, S. (2002). The NIMH blueprint for change report: Research priorities in child

and adolescent mental health. Journal of the American Academy of Child and Adolescent

Psychiatry, 41(7), 760-767.

Jensen, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy

of Child and Adolescent Psychiatry, 42(5), 527-530.

Morgenstern, J. (2000). Effective technology transfer in alcoholism treatment. Substance Use &

Misuse, 35(12-14), 1659-78.

Simpson, D. D. (2002). A conceptual framework for transferring research to practice . Journal of

Substance Abuse Treatment, 22(4), 171-182.

Trochim, W. M., K, Cook, J. A., & Setze, R. J. (1994). Using concept mapping to develop a conceptual

framework of staff's views of a supported employment program for individuals with severe mental

illness. 62(4), 766-775.

Documents

19th Annual RTC Conference Presented in Tampa, …rtckids.fmhi.usf.edu/rtcconference/handouts/pdf/19...19th Annual RTC Conference Presented in Tampa, February 2006 1 CASRC Multiple