Evidence-based Practice for Applied Behavior Analysts: Necessary or Redundant

Evidence-based Practice for Applied Evidence-based Practice for Applied Behavior Analysts: Necessary or Behavior Analysts: Necessary or

RedundantRedundant

Ronnie Detrich, Wing Institute

Tim Slocum, Utah State University

Teri Lewis, Oregon State University

Trina Spencer, Northern Arizona University

Susan Wilczynski, Ball State University

Goals for TodayGoals for Today

Describe basic concepts of evidence-based practice.

Describe integrated decision making framework.

Two Ways of Thinking about EBPTwo Ways of Thinking about EBP

An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)

Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)

Two Ways of Thinking about Two Ways of Thinking about EBPEBP

Using same term (EBP) to describe two different constructs creates confusion:

Empirically supported treatments (EST)-interventions that meet defined standards of quality and quantity.

Evidence-based practice-decision making process.

Evidence-based Practice and Evidence-based Practice and Applied Behavior AnalysisApplied Behavior Analysis

EBP is framework for guiding decisions of practitioners.

Decisions are based on the integration of:

Best available evidence

Client values and context

Professional judgment

Consistent with foundational principles of applied behavior analysis:

Data-based decision making

Consideration of client values

Considerations of contextual fit

Commitment to research-based treatment

Evidence-based Practice and Evidence-based Practice and the Research-Practice Gapthe Research-Practice Gap

Across disciplines, great concern about the discrepancy between what is known from research about effective treatments and the interventions practitioners routinely employ.

EBP is basis for closing the gap.

How Does EBP Narrow the How Does EBP Narrow the Research to Practice Gap?Research to Practice Gap?

Practitioners must make decisions “now” even when research evidence is absent or incomplete. What is to be the basis for decisions?

Decisions informed by best available evidence allows practitioners to: Select

Adapt to fit local circumstances

Modify on the basis of progress monitoring data

These decisions require professional judgment.

Dilemma for PractitionersDilemma for Practitioners

Practitioners must make many decisions daily.

EBP assumes there is evidence for all decisions and that the relevant evidence is accessible.

How do practitioners incorporate evidence into all decisions?

The challenge of The challenge of Best available EvidenceBest available Evidence

GoalGoal

The best available evidence should pervade the practice of Applied Behavior Analysis.

What is the “Best Available Evidence” for ABA practice?

How do we systematically identify it?

Best Available EvidenceBest Available Evidence What do we mean by “best”?

Quality: research methods and outcomes

Relevance: close match with our applied question in terms of:

• Participants

• Treatment

• Outcomes

• Context

Amount: number of participants, studies, investigators

Best available evidenceBest available evidence

Quality

High

Low

Low High

Better e

vidence

Relevance(P, T, O, C)

Empirically Supported Treatments

Empirically Supported Treatments

Best Available EvidenceBest Available Evidence

What do we mean by “best available”?

We should use the best of what is available,

• This may mean using extremely high quality evidence

• Or it may mean using evidence that is less certain.

“Unlimited skepticism is equally the child of imbecility as implicit credulity.”

Dugald Stewart

“Unlimited skepticism is equally the child of imbecility as implicit credulity.”

Dugald Stewart


Quality

High

Low

Low High



Quality

High

Low

Low High

Empirically-Supported Treatment

Empirically-Supported Treatment


Need to GeneralizeUncertainty

The practical problemThe practical problem

Practitioners must often make decisions with insufficient empirical support.

What are they to do?

Make the best possible inferences from imperfect evidence?

Make decisions without using systematic evidence?

The practical problemThe practical problem

If Evidence-Based Practice of ABA is to be a pervasive model for professional decision-making…

then we need ways to identify the best available evidence when the evidence is imperfect.

Accessing the Best Available Accessing the Best Available EvidenceEvidence

1. Systematic reviews – The foundation

Identifies empirically supported treatments

2. Alternative types of review

Improve our ability to glean recommendations from imperfect literature

3. Other units of practice - beyond the package

Using what we know about intervention elements and principles

4. Progress Monitoring

The best evidence about what works for this particular case

1. Systematic reviews1. Systematic reviews

Systematic ReviewsSystematic Reviews

• Systematic EBP review (e.g., WWC, BEE, NAC)

Establish standards for:

Identifying research base

o Participantso Interventionso Comparisonso Outcomeso Settings

Quality of evidence

Quantity of evidence

Unit typically limited to “programs” (treatment packages)

Systematic ReviewsSystematic Reviews

Reduced bias

Transparency

Objectivity

Rigorous methods

Reduced risk of false positives

Exclusive reliance on high quality evidence

Often fail to identify sufficient evidence

Not informed by lower quality evidence

Higher risk of false negative.

Strengths Limitations

2. Alternative types of 2. Alternative types of reviewsreviews

a. What Works Practice Guidesa. What Works Practice Guides

Practice guide system:

Provide:

expert recommendations and

specific level of supporting evidence.

Allows broader generalization from specific studies

Allows for expert recommendations on topics where evidence is spare

a. What Works Practice Guidesa. What Works Practice Guides

Expert interpretation of research

Includes “level of evidence” ratings – based on systematic review.

Expert interpretation may introduce bias & increase uncertainty


b. Best Practice Panelsb. Best Practice Panels

• Best practice panel

Panel selection

Organization selects “expert” panelists

Panel can be broadly or narrowly constructed

Key to validity

o Face validity o Functional aspects of validity

Recommendations based on group consensus

b. Best Practice Panelsb. Best Practice Panels

May include diverse perspectives:

Researcher

Practitioner

Consumer

Allows for interpretation of research

Tend to include factors other than research

social validity?

contextual fit?

May lack transparency:

Selection of panel

Criteria for “best practice”: politics or science?

Tend to include factors other than research

Bias?


c. Narrative Reviewsc. Narrative Reviews

• Narrative review

E.g., APA Handbook of Behavior Analysis;NASP Best Practices;

• Experts review research base to establish recommendations

• Allows experts to draw on wide range of evidence

• Allows for expert interpretation of patterns of relevant findings

c. Narrative Reviewsc. Narrative Reviews

Allows for broad generalization from specific studies to implications for practice.

Can incorporate many sources of evidence and logic/theory

No methodology to reduce bias.

Selecting experts

Relevant research base

Criteria for “best practice”

Strength of evidence rating


3. Other units of practice3. Other units of practice

What is a “treatment”?What is a “treatment”?

The best available evidence can validate:

← Comprehensive schoolwide systems

← Multi-component instructional or behavior packages

← Specific components/elements/kernels

← Principles of behavior and learning

a. Practice Elements and a. Practice Elements and KernelsKernels

ExamplesDifferential praise, self-monitoring, frequent

student responses with corrections

How they are validatedKernels

• Kernels are implemented, outcomes are measured.

Practice elements

• Effective treatment packages are analyzed.

• Practice elements are components included in most effective packages.

a. Practice Elements & Kernelsa. Practice Elements & Kernels

More flexible than multi-component programs.

May be used to assemble custom interventions directed at specific problems

Can supplement information on multi-component packages

A set of effective components may not produce an effective package.

Assembling package is time-consuming and requires high level of skill.


b. Principles of Learning and Behaviorb. Principles of Learning and Behavior

Examples

Differential reinforcement, extinction

Principles of using examples and non-examples

How they are validated

Numerous studies across wide variety of populations and contexts.

b. Principles of Learning and Behaviorb. Principles of Learning and Behavior

The most basic and flexible guides to intervention.

Apply to huge range of educational problems including modifications & adaptations.

Can supplement evidence on packages and components

Do not provide specific interventions. Principles must be applied – this process is uncertain.

Application requires extremely high level of expertise.


4. Progress monitoring4. Progress monitoring

4. Progress Monitoring4. Progress Monitoring

Progress monitoring can validate:

The specific treatment (as modified)

with the specific client

on the specific outcomes

in the specific context

No other evidence can substitute.

4. Progress Monitoring4. Progress Monitoring

Strengths

The best evidence on whether this particular program is working.

Provides basis for data based decision making

Limitations

Does not help:

initial selection of treatments

select modifications

ConclusionConclusion

The best available evidence to support pervasive evidence-based practice can be derived from:Empirically supported treatments

Evidence from alternative types of reviews

Evidence on alternative units of practice

And should always include progress monitoring.

Client Values and ContextClient Values and ContextTeri Lewis

Oregon State University

Beyond ESTBeyond EST

Even when there are EST, practitioners may not choose these interventions and/or implement them when they are recommended.

In education innovations come and go in 18-48 months (Latham, 1988).

So, if we have the right So, if we have the right answers why aren’t EST answers why aren’t EST adopted and sustained?adopted and sustained?

Face Validity

Client-values and context need to be included and respected

Historical PerspectiveHistorical Perspective Baer, Wolf & Risley (1968)

Immediate and important change in behavior that has practical value as determined “…by the interest which society shows in the problems.” (p. 92)

Wolf (1978) Social significance of the goals.

Societal value

Social appropriateness of the procedures. Treatment acceptability

Social importance of the effects. Consumer satisfaction

““Client”Client”

Individual(s) who are invested in outcomes and/or are critical to the behavior change process (e.g., Baer et al, 1968; Heward et al, 2005)

Individual who is the focus of the behavior change

Parents and family members, teachers, mentors, colleagues, employers

Organization, society

• In contrast to EBP, BACB (2010) relies on a narrow definition

• Individual receiving services from a behavior analyst

• Decision about who is a client based on:

• Ethics

• Acceptability & Fidelity

• Effectiveness

• Maintenance and sustainability

Context Context

Just as we focus on behavior within an environmental context , we need to consider the context for selection and implementation of interventions

Contextual-fit (e.g., Albin et al 1996)

Values, skills, goals and stressors of the implementors and those impacted by the target behavior

Implementation and Implementation and AcceptabilityAcceptability

Motivating operations such as values, goals and stressors

Compatibility with other aspects of the context (e.g., routines, resources, policies)

Reinforcement and punishment associated with implementation

Clients ultimately decide whether the intervention effects are judged successful(Wolf, 1978)

Strain, Barton & Dunlap (2012) assert that incorporating client-values successfully informs decision-making:

Design of service delivery

Technical support to key implementors

When to fade intervention components

Identification of unanticipated events

Focus of future research needs

Ethical PerspectiveEthical Perspective

Three basic and fundamental questions (Cooper, Heron & Heward, 2007)

What does it mean to be a behavior analyst?

What is the right thing to do?

What is worth doing?

• Social Validity

• Client Values

SummarySummary

Including client values into assessment, intervention selection and implementation

Respects all individuals perspectives

Increases acceptability

Improves decision-making

Increases fidelity and sustainability

Specifically, client values inform selection of treatment and the methods of treatment

Art of behavioral interventions

But, most importantly, it allows behavior analysts the opportunity to implement our EST in applied settings and to produce important changes in behavior that have practical value

Professional judgmentProfessional judgment

Professional JudgmentProfessional Judgment

Process by which the best available evidence is applied to individual cases in specific contexts.

An evidence-based practice framework provides an opportunity for behavior analysts to understand the role of professional judgment so that we can ensure all behavior analysts are wise decision makers and ultimate improve our effectiveness.

Evidence-based Practice Evidence-based Practice FrameworkFramework

Spencer, Detrich, & Slocum, 2012

Client and Context

Behavior analysts use Behavior analysts use professional judgment…professional judgment…

…when formulating a practical question based on the presenting needs of the client and context.

Hudson engages dangerous behaviors such as climbing houses to get attention and because climbing is fun. What treatment should I use?

Client and Context

…to determine what evidence to look for, where to look for it, and to make sense of it in relation to the practical question and client.

Conduct a search for a systematic review or studies on treatments for dangerous behaviors that involve attention and automatic reinforcement functions.

Client and Context


…to weigh the quality and relevance of the currently available scientific knowledge in relation to the client and context to select a treatment.

Differential reinforcement of other behaviors and a regular routine of appropriate replacement activities.


Client and Context

Client and Context

…to adapt a selected treatment to be suitable for the specific client and context based on the best available evidence.

For every day that Hudson does not engage in dangerous behaviors, he earns 30 minutes of a special activity. In addition, he participates in a rock climbing club for children every week.


Client and Context

…to implement and monitor a selected treatment that was adapted for a specific client and context, based on the best available evidence.

Hudson can monitor his dangerous behaviors and record their absence on a calendar. He can report to his mother every evening and request 30 minutes of a special activity.


Client and Context


…to determine when a socially meaningful outcome has been achieved, based on the specific client and the best available evidence.

Spencer, Detrich, & Slocum, 2012


Behavior analysts use professional judgment in their practice every day.

Importantly, these judgments are a necessary part of Applied Behavior Analysis.

The pervasiveness of professional judgment suggests that we should be very skilled at this activity.

Bases of Professional JudgmentBases of Professional Judgment

Behavior analysts can draw from the available scientific bases to strengthen professional judgments.

Principles of learning and behavior

Ongoing progress monitoring


Principles of behavior, grounded in decades of experimental and applied research, serve as a foundation for judgments made by behavior analysts.

Behavior analysts can identify the principles that contribute to specific decisions (e.g., discrimination, differential reinforcement, extinction, schedules of reinforcement).


Most of our decisions can and should be informed by close and continual contact with relevant outcome data.This does not mean that we do not use

professional judgment.

It means our judgments are data-based.

Ongoing progress monitoring is THE best evidence of what works with specific individuals in specific contexts regardless of the strength of the empirical research.


Given professional judgment is ubiquitous, necessary and it has a scientific basis, it serves our profession to embrace it, develop it, and promote it.

Doing so, has implications for training and professional development.

Training ProfessionalsTraining Professionals

A great deal of training is necessary to build the expertise required to be effective behavior analysts.

The BACB training sequence is designed to ensure that behavior analytic practitioners are prepared to make good judgments.

Training ProfessionalsTraining Professionals

Coursework is important, but professional judgment is primarily shaped through supervised field experiences.

Guided or mentored experiences are necessary to teach the subtle discriminations required to be an effective behavior analytic practitioner.

Take Home PointsTake Home Points

Professional judgment is pervasive in our practice and necessary for behavior analysts to be effective.

Professional judgment can be strengthened through an explicit recognition of its role, regular contact with empirical research, careful integration of principles and tactics, and ongoing measurement of relevant outcomes.

Professional judgment can be developed and enhanced via quality preservice training, supervised field experiences, and real world contingencies.

RecommendationsRecommendations

Adopt a decision-making framework to help guide practitioners to make judgments based on the best available evidence and client and context.

Organize our scientific knowledge so that principles and tactics are more accessible to practitioners who rely on them as a basis for professional judgment.

Infuse behavior analytic training programs with professional decision making.

Conclusions & Contingencies for Conclusions & Contingencies for EBP of ABAEBP of ABA

Formerly named: Contingencies and Evidence-Based Practice: Are we seeking punishers for

our scientists and practitioners?

Two Ways of Thinking about EBPTwo Ways of Thinking about EBP

An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)

Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)

EBP of ABAEBP of ABA

EBP is process for guiding decisions of practitioners.

Decisions are based on:

Best available evidence

Client values and context

Professional judgment

An EBP to ABA framework promotes translational methods at the practitioner level.

Translational MethodsTranslational Methods

Translational MethodsTranslational Methods

EBP of ABAEBP of ABA

Consistent with foundational principles of ABA

Data-based decision making

Consideration of client values

Considerations of contextual fit

Consonance of Behavior Analysis and EBPBehavior Analysis Evidence-Based Practice

Analytic BAE* – goal: establish a functional relationship based on local data collection and progress monitoring

Technological BAE* – methods that are sufficiently well defined to allow for treatment fidelity & replication

Conceptually Systematic

BAE* – well documented interventions are understood as examples of a unifying base

Professional Judgment – selecting/adapting tx based on an integrated system in lieu of ‘bag of tricks’

Applied Client values and context* – improve quality of life targeting socially significant behavior

Effective Client values and context* – effective change of practical and social significance

Generality BAE & Client values and context* – programming across time, settings, people

Behavioral BAE* – must be measureable, precise, reliable

Baer, Wolf, & Risley (1968)*BAE & Client values/Context always involve professional judgment

Consonance of ABA and EBPConsonance of ABA and EBP

Behavior Analysis Certification Board

Best Available Evidence

• Identify interventions based on “best available scientific evidence”

Client Values & Context

• Select treatments based on a range of variables including but not restricted to: client preferences; client’s current repertoires; supporting environments including the constraints on those environments; social validity

Professional Judgment

• Making determinations about treatment effectiveness based on graphical display of data (in various formats)

Implications of EBP of ABA: Implications of EBP of ABA: PracticePractice

Help provide an explicit rationale for decision-making that is consistent with the rationale forwarded by virtually every other discipline.

Decision-making framework is:

Transparent about how information regarding evidence is evaluated

Clear about why evidence is most appropriate for a given case

Cogent regarding how client values and contextual factors influence treatment selection (and the decision to retain/alter a treatment)

Highlights critical analytic thinking and why this analytic process should be applied to all treatments considered


Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholdersClient and contextual factors

• Define the goals for treatment and acceptable methods for intervention

• Serves to prompt us to focus on these factors that have always been held important to our field

• To meet this goal: practitioners must be prepared to discuss the goodness of contextual fit

• By referring to these factors in practice, we undermine the fallacious argument that we are ‘cold’ in our decision-making


Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholdersProfessional judgment

• An EBP of ABA framework gives us a way to discuss the expertise we bring to the table

• Common framework and language adopted by other fields but allows us to highlight our unique strengths in treatment evaluation on an individual basis

• Allows us to reject practitioners who argue EBP involves following a ‘list’

Implications of Ignoring Implications of Ignoring EBP of ABA: PracticeEBP of ABA: Practice

Portrayed as ‘ignorant’ of the most pervasively adopted framework across disciplines

Our professional practice may be undermined with: Practitioners in other disciplines

Clients & Families

Administrators

Insurance Reimbursement

Implications of Ignoring Implications of Ignoring EBP of ABA: ScienceEBP of ABA: Science

Federal agencies and funding

Institute for Educational Sciences

NIMH

DHHS

Future DirectionsFuture Directions

By highlighting ‘best available evidence’ we identify that further refinements are necessary

As highlighted by Smith (2013), we must continue to validate practice elements or kernels


Dissemination remains the great challenge of all fieldsHow do we get our information about

effective treatments organized well enough that it is accessible to practitioners?

How do we make decisions when the best available evidence is not ‘good enough?’

How do we set guides for optimal care for behavior analysts that will improve the overall quality of the services we provide?


Training of new behavior analysts

The EBP of ABA framework suggests the decision-making process is the cornerstone of good practice – not a list of treatments

Shifting greater emphasis on client values and contextual variables

Requiring a rationale for how decisions are made by students in all training activities and providing feedback will enhance professional judgment (and thus practice)

Documents

Evidence-based Practice for Applied Behavior Analysts: Necessary or Redundant