An Analysis of Evidence-Based Practices in the Education ...daddcec.org/Portals/0/CEC/Autism_Disabilities/Research/Publications/... · An Analysis of Evidence-Based Practices in the

An Analysis of Evidence-Based Practices in the Education andTreatment of Learners with Autism Spectrum Disorders

Michael R. MaytonWest Virginia University

John J. WheelerWestern Michigan University

Anthony L. MenendezCleveland State University

Jie ZhangThe College at Brockport: State University of

New York

Abstract: Horner et al. (2005) present a review substantiating how single-subject research methodology can beutilized to determine whether interventions are evidence-based practices (EBPs). The current study utilized theHorner et al. research piece to: (a) systematically identify a set of quality standards for the evaluation ofsingle-case research methodology used with learners with autism spectrum disorders (ASD), (b) operationalizethese standards for evaluators, (c) investigate three additional quality indicators related to external validity(multiple studies, locations, and researchers), (d) create a protocol for evaluators, and (e) gather and analyzedata from studies that meet a set of predefined criteria. Published in seven journals across ten years, identifiedstudies (N � 160) were subjected to an analysis across 23 EBP quality standards that revealed increasingcompliance with EBP standards over time yet persistent and pervasive difficulty in adequately satisfying at leastfive indicators integrally tied to external validity.

The term EBP originated in the field of med-icine as early as the 1970s (Odom et al., 2005)and has become an integral part of medicaleducation. The intent of this practice was tominimize the gap between research and prac-tice with the end goal being directed towardthe use of scientific evidence as the method ofchoice for physicians in the diagnosis andtreatment of illness. The movement for evi-dence-based practices (EBP) in the field ofeducation was introduced with the passage ofthe No Child Left Behind (NCLB) Act of 2001(Odom et al.). This significant piece of legis-lation emphasized the use of science-basedmethods by teachers in their classrooms andwas aimed at restructuring the educationalsystem with increased standards and account-ability as its cornerstones (Simpson, 2005).The role of EBP in the field of special educa-tion has certainly been influenced in part bythe NCLB Act, yet the field has had a rich

history of research-based methods in the edu-cation and treatment of persons with disabili-ties (Odom et al.). Many of the early investi-gations in the field were derived from casestudies that were generated from the tradi-tions in the field of psychology. As the field ofspecial education progressed, we witnessedthe use of more elaborate research designs,including the use of single-subject experimen-tal designs that resulted in more experimen-tally valid research. Yet, despite this rich learn-ing history as a field, special education, likethe field of education in general, has beenchallenged with a significant gap between re-search and practice. In other words, what weknow to be effective in terms of teaching andlearning based on research findings is oftennot employed with learners in the classroom.This gap is also evident with respect to theeducation and treatment of learners with au-tism, which has resulted in the term EBP gain-ing a great deal of recent notoriety with re-spect to the treatment of autism, especiallygiven the increasing prevalence rates of chil-dren being diagnosed with the condition.Now, 1 in every 110 children is estimated to bediagnosed with autism (Centers for Disease

Correspondence concerning this article shouldbe addressed to Michael R. Mayton, West VirginiaUniversity, Department of Special Education, 508-CAllen Hall, Morgantown, WV 26506-6122.

Education and Training in Autism and Developmental Disabilities, 2010, 45(4), 539–551© Division on Autism and Developmental Disabilities

An Analysis of Evidence-Based Practices / 539

Control, 2009). Children and families affectedby autism represent a very vulnerable popula-tion, and understandably families seek out avariety of treatments for their children in aneffort to minimize the lifelong effects of au-tism. Conversely, the availability of non-evi-dence-based treatments for children affectedby autism are substantial in number and oftenpose a threat to the well-being of these chil-dren and their families (Simpson), thus theimportance of identifying evidence-base treat-ments within research to better inform prac-tice. To better understand EBP, one must un-derstand at least one prominent way in whichthe term has already been operationalized forlearners with autism spectrum disorders(ASD).

The National Professional DevelopmentCenter (NPDC) on Autism Spectrum Disor-ders (2009) defines EBP in the context ofpeer-reviewed research published in profes-sional journals. The NPDC’s position is that,in order to be termed an EBP, an interventionmust be presented within studies published insuch journals and constitute a research basesubstantiated by: (a) two high quality experi-mental or quasi-experimental group designstudies, (b) five high quality single- subjectdesign studies conducted by three differentresearchers, or (c) one high quality random-ized or quasi-experimental group design studyand three high quality single-subject designstudies conducted by at least three differentresearchers.

The importance of selecting and usingEPBs for learners with ASD cannot be over-stated, especially given (a) the prevalence ofASD is increasing at an alarming rate, (b) thehigh-stakes nature of intervention outcomesfor children with ASD and their families, and(c) the history of questionable treatments thathave been marketed to children and familiesaffected by autism. These are obvious justifi-cations for the importance of EBP in the re-search literature and subsequent practice.However, a gap not only exists between re-searchers and practitioners but in how re-search is conducted in terms of quality assur-ance. Issues such as treatment fidelity (i.e., ifinterventions are implemented in a reliablemanner, as designed) have plagued the liter-ature. One study conducted by Wheeler, Bag-gett, Fox, and Blevins (2006) reviewed behav-

ioral treatment studies in the field of autismfrom 1993–2003 across nine peer reviewedjournals specializing in such studies. The re-sults from this investigation revealed that ofthe 60 studies reviewed, only 11 studies as-sessed treatment integrity and provided anadequate operational definition of the inde-pendent variable. This fact is startling givenhow research can potentially impact practice.

The purpose of this study was to provide ananalysis of research relative to EBP with learn-ers identified with ASD, using the standardsfor evaluating single-subject research as rec-ommended by Horner et al. (2005). Implica-tions for practice and research are provided.

Method

The Evidence-Based Practice Construct

Horner et al. (2005) present a review andrationale substantiating the foundation thatexisting scientific knowledge and procedureregarding single-subject research methodol-ogy can be utilized to determine whether in-terventions are EBPs. The authors discuss, de-fine, and provide examples of eight qualityindicators within single-subject research andadd further recommendations for strengthen-ing the external validity of this research. Thecurrent study utilized the Horner, et al. re-search piece to: (a) systematically identify a setof quality standards for the evaluation of sin-gle-case research methodology used withlearners with ASD, (b) operationalize thesestandards for evaluators, (c) investigate threeadditional quality indicators related to exter-nal validity, (d) create a protocol for evalua-tors, and (e) gather and analyze data fromstudies that meet a set of predefined criteria.

Journals and Article Selection Criteria

Seven journals were chosen for the reviewbased on their prominence in the field ofsingle-subject research conducted with per-sons with ASD. Selected journals were as fol-lows: (a) Education and Training in Developmen-tal Disabilities (ETDD), (b) Education andTreatment of Children (ETC), (c) Focus on Au-tism and Other Developmental Disabilities (FO-CUS), (d) Journal of Applied Behavior Analysis

540 / Education and Training in Autism and Developmental Disabilities-December 2010

(JABA), (e) Journal of Autism and DevelopmentalDisorders (JADD), (f) Journal of Positive BehaviorInterventions (JPBI), and (g) Research in Devel-opmental Disabilities (RDD).

In order to be included in the currentstudy, research had to adhere to four maincriteria. First, all participants included in astudy had to have a diagnosis that is some-where on the autism spectrum of disorders.Articles utilizing “mixed” samples were re-jected. A mixed sample was considered to beone in which one or more of the participantsin a study had a sole diagnosis other thanautism. For example, studies were rejected ifthey utilized samples that included partici-pants with a sole diagnosis of mental retarda-tion as well as participants with autism. How-ever, studies were accepted in whichparticipants were diagnosed with autism andcomorbid disorders (e.g., one or more indi-viduals were diagnosed with both autism andmental retardation). In addition, all partici-pants had to be from 3 to 21 years of age, andeach study had to involve a behavioral treat-ment. A behavioral treatment was defined asan intervention-based study aimed at the de-velopment of skills or behaviors within partic-ipants using a single-case experimental de-sign. Last, all articles had to have beenpublished within the predefined, ten-yearrange (2000–2009).

Standards and Operational Definitions

Twenty-three standards were distilled directlyfrom the Horner et al. (2005) discussion ofquality indicators found within single-subjectresearch. Horner, et al. also recommendedthat external validity be strengthened“through systematic replication of effectsacross multiple studies conducted in multiplelocations and across multiple researchers”(p.171), hence the current examination of:(a) most prolific authors, (b) locations whereresearch is being conducted, (c) number ofparticipants per study, and (d) the number ofstudies within each category of independentvariable. Table 1 provides a complete list ofthe indicators and standards, as well as exam-ples of operational definitions.

Raters, Ratings, and Interobserver Reliability

The authors of the current study served asraters due to their expertise and applied ex-perience in the area of single-subject researchdesign. The first author provided the otherauthors with: (a) journal assignments, (b) theHorner et al. (2005) article, (c) article selec-tion criteria, and (d) the data collection pro-tocol (including general instructions, item de-scriptions, operational definitions, andindexed references to specific content withinthe Horner, et al. article). Raters were given atwo-week period for review of materials andclarification of any questions, and during thisperiod the first author engaged raters in aseries of phone and face-to-face conversationsto check for understanding of the protocol,associated materials, and procedures.

Raters were instructed to evaluate the 23standards across each study in terms of “ac-ceptable” or “not acceptable.” In order for astudy to be rated “acceptable” within a partic-ular standard, the study had to present evi-dence fully satisfying each part of the opera-tional definition for that standard. No partialratings were allowed. For instance, in order torate a study as one that describes the indepen-dent variable (IV) with replicable precision,raters had to find adequate evidence thatmatched the operational definitions for: (a)systematic manipulation of the IV, (b) IV un-der the control of the experimenter, and (c)measurement description and reporting oftreatment integrity data. Failure in one ormore of these areas would result in a rating of“not acceptable” for this indicator.

Once all initial ratings were completed,20% of the articles chosen for review withinthe current study (n � 32) were randomlyselected and distributed to the first and sec-ond authors, making sure that the authorsreceived only articles that had been previouslyrated by other reviewers. The studies werethen independently read and rated, andagreement was calculated by dividing thenumber of agreements plus disagreementsand multiplying by 100. The percentage ofagreement across all ratings averaged 93.5%(23 ratings for each of 32 studies � 736 indi-vidual ratings).


Results

General Study Characteristics

Articles from JABA, JPBI, JADD, and FOCUSmade up the majority (79%) of the 160studies included in the current study. Acrossall included articles, no study exceeded 10participants, with the next highest maxi-mum being 7 participants. Three journalspublished studies that were conducted

within all six of the major United Statesgeographical areas (as well as outside theU.S.), and across all journals multicompo-nent independent variables (IVs) were themost commonly found. IVs that were com-posed of treatment packages, or those madeup of more than one significant component,were labeled “multicomponent.” This wasdone to avoid the potential confound ofincluding studies within a particular evi-

TABLE 1

Quality indicators in single-subject research, rating standards, and example operational definitions.

Quality Indicator (Horner, etal., 2005)

Rating Standard Example Operational Definition

Description of participantsand settings

1. Description ofparticipants

2. Participant selectionprocess

3. Description ofsetting

Description of participants (p. 167): Participants aredescribed in sufficient detail to allow others to selectindividuals with similar characteristics.

1. The specific disability and the specific instrumentand process used to determine the disability areidentified.

2. Specific characteristics directly pertaining to theDV and IV are reported (e.g., if the DV and IVdeal with some aspect of communication, priorlevels of communication ability should bespecified).

Dependent variable 4. Operationaldefinition

5. Measurementprocedure

6. Valid and precise7. Repeated

measurement8. Interobserver

agreement9. Social significance

Interobserver agreement (p. 167): IOA is reported toa sufficient extent and meets minimal or higherstandards.

1. IOA is assessed across each variable, participant,and condition within the study.

2. These data meet or exceed minimal standards:IOA � 80% and Kappa � 60%.

Independent variable 10. Description11. Systematic

manipulation12. Implementation

fidelity

Implementation fidelity (a.k.a., treatment integrity,treatment fidelity) (p. 168): The fidelity of IVimplementation is fully documented.

1. Overt measurement of implementation fidelityreportedly occurred.

2. These data were reported.Baseline 13. Pattern of

responding14. Description of

conditions

Pattern of responding (p. 168): Measurement of theDV during the baseline condition demonstrates apattern of responding that allows prediction of futureresponding.

1. There are at least five data points displayed inthe initial baseline condition (see below forexceptions).

2. The data display no discernable trend, or theydisplay a trend in the opposite direction thanthat predicted by the intervention.


dence-base grouping (e.g., “social stories”)that have other significant intervention fea-tures that could be responsible for some orall of the reported results. See Table 2 fordetails regarding these characteristics.

Indicators of External Validity

Replication of effects. Most of the articlesidentified (80%) had from one to three par-ticipants, as expected within single-subject re-

TABLE 1 (Continued)

Quality Indicator (Horner, etal., 2005)

Rating Standard Example Operational Definition

Experimentalcontrol/Internal validity

15. Demonstratedeffect

16. Controls for threats17. Demonstration of

experimental control

Demonstration of experimental control (p. 168):Experimental control is demonstrated to the extent that afunctional relationship is indicated between themanipulation of the IV and the change in the DV.

1. The design documents three demonstrations ofexperimental effect at three different points in timewith a single participant, or across differentparticipants.

2. Documentation of experimental control is achievedthrough: (a) the introduction and withdrawal of theIV; (b) the staggered introduction of the IV atdifferent points in time; or (c) the iterativemanipulation of the IV (or levels thereof) acrossobservation periods.

External validity 18. Replication Replication (p. 171): Experimental effects are replicatedacross relevant variables in order to increase externalvalidity.Experimental effects are demonstrated across atleast three different participants, settings, sets ofmaterials, and/or behaviors.

Social validity 19. Social importanceof the DV

20. Magnitude ofchange

21. Practical/cost-effective

22. Enhancement

Enhancement (p. 172): Social validity is enhanced bysocial importance, typical application, and the positivereport of intervention agents.

1. DV(s) were selected that have a high “socialimportance,” or emphasis on helping an individualeffectively operate in typical/needed contexts inwhich other people also operate.

2. It was demonstrated that the intervention(s) couldbe implemented with fidelity by typical interventionagents (e.g., parents) in typical contexts acrossmeaningful periods of time.

3. Typical intervention agents report interventionprocedures to be feasible within available resources,effective, and useful after formal intervention isdiscontinued.

Research question(s) 23. Appropriateness ofresearch question(s)for single-subjectresearch methods

Appropriateness of research question(s) (p. 172):Single-subject methodology is used to investigate functionalrelations between the IV(s) and DV(s), pertaining to theperformance of a specific individual.

1. Each question deals with the effects that systematicmanipulation of an IV (or component of an IV) hason one or more DV(s).

2. Each question is concerned with/focuses upon theperformance of specific individuals under a givenset of conditions.


search studies. However, it was demonstratedthat a sole single-subject study could effec-tively utilize as many as 10 participants,though this occurred in only one study that fit

the search criteria (see Figure 1 for percent-ages within each category).

In addition to reviewing the number of par-ticipants within studies, replication of effects

TABLE 2

General study characteristics.

Journal Number andPercent ofStudies

Identified(N � 160)

MaximumParticipants

AcrossStudies

GeographicalLocations

Represented

GeographicalLocations

Unrepresented

Most CommonIndependentVariable(s)

ETDD 12 (8%) 6 Midwest Atlantic states multicomponentMountain states Pacific statesNon-U.S. SouthwestSouth

ETC 6 (4%) 4 Atlantic states Midwest multicomponentPacific states Mountain statesSouth Non-U.S.

SouthwestFOCUS 22 (14%) 6 Atlantic states social stories,

multicomponentMidwestMountain statesNon-U.S.Pacific statesSouthSouthwest

JABA 50 (31%) 7 Atlantic states multicomponent,prompting/reinforcementMidwest

Mountain statesNon-U.S.Pacific statesSouthSouthwest

JADD 26 (16%) 10 Atlantic states multicomponent, socialstoriesMidwest

Mountain statesNon-U.S.Pacific statesSouthSouthwest

JPBI 29 (18%) 4 Atlantic states Mountain states video modeling,multicomponentMidwest Southwest

Non-U.S.Pacific statesSouth

RDD 15 (9%) 6 Atlantic states Mountain states prompting/reinforcement,imitation trainingMidwest

Non-U.S.Pacific statesSouthSouthwest


was examined by categorizing the types of IVsutilized across studies. The formation of cate-gories was based on the following decisionguidelines: (a) there had to be at least twostudies with the same IV, and (b) the IV couldnot be classified as “multicomponent” (as pre-viously described). The top four IV categoriesin terms of frequency across the ten-year ex-amination period were: (a) multicomponent,

(b) prompting/reinforcement, (c) social sto-ries, and (d) video modeling, comprising 64%of the articles across all categories. Table 3provides a complete listing of the categoriesthat emerged using the decision guidelines.

Multiple locations. Within groupings of thesix major U.S. geographical areas and non-U.S. locations, the largest numbers of studiesthat fit the search criteria were produced inthe South and Atlantic States regions (to-gether, 44% of all studies). The regions pro-ducing the fewest studies were the Southwestand Mountain States (together, 12% of allstudies). Figure 2 graphically displays percent-ages across these regions.

Multiple researchers. Using the criterion ofidentifying first-authors who published 3 ormore studies within the ten-year examinationperiod, only six authors were categorized as“most prolific.” Approximately 14% of the ar-ticles identified for use within the currentstudy (n � 22) were produced by these sixauthors, with one author producing 5 studies(3.1% of all studies identified), two authorsproducing 4 studies each (2.5 % each), andthree authors producing 3 studies each (1.8%each).

Figure 1. Percent of studies by number ofparticipants.

TABLE 3

Number of studies by independent variable (IV).

Independent Variable (brief descriptor) Number of Studies:2000–2009

1. Activity schedule...................................................................................................................... 42. Computer-based instruction (CBI) ....................................................................................... 33. Choice ...................................................................................................................................... 34. Differential observing responses (DOR) .............................................................................. 25. Discrete trial training (DTT)................................................................................................. 36. Functional communication training (FCT) ......................................................................... 77. Imitation training.................................................................................................................... 58. Language instruction.............................................................................................................. 39. Modeling.................................................................................................................................. 210. Multicomponent ................................................................................................................... 3511. Picture Exchange Communication System (PECS)........................................................... 412. Peer-mediated/assisted interventions.................................................................................. 513. Prompting/reinforcement ................................................................................................... 2314. Scripts..................................................................................................................................... 315. Social stories .......................................................................................................................... 1416. Strategy instruction............................................................................................................... 317. Video modeling..................................................................................................................... 13

Note: IVs that appeared in only one study across the ten-year period were not included in this analysis.


Standards Ratings

Standards ratings by IV. The four IV cate-gories with the highest percentages of “notacceptable” ratings were: (a) discrete trialtraining (41%), (b) choice (28%), (c) videomodeling (22%), and (d) functional commu-nication training (20%). The four IV catego-ries with the highest percentages of “accept-able” ratings were: (a) strategy instruction(96%), (b) activity schedules (95%), (c) com-puter-based instruction (93%), and (d) imita-tion training (92%). Figure 3 provides agraphical representation of these percentagesacross all IV categories identified, and Table 4shows percentage groupings across individualIVs.

Figure 2. Percent of studies by geographicallocation.

Figure 3. Percentage of standards ratings by IV.


TA

BL

E4

Grouped

standardsratings

acrossIV

s.

Activity

sched.C

BI

Choice

DO

RD

TT

FCT

Imitation

train.L

anguageinstr.

Modeling

Multicom

ponentPEC

SPeer-m

ed./asst.

Prompting/R

.Scripts

SocialStories

StrategyInstr.

VideoM

odeling

Standard1

�V

VV

��

�V

��

�●

V�

�

2�

�V

V●

��

V●

●�

●V

V●

3�

VV

��

��

●V

��

V�

�

4�

��

��

��

��

��

��

��

��

5�

��

��

��

��

��

��

��

��

6�

��

��

��

��

��

��

��

��

7�

��

��

��

��

��

��

��

��

8�

��

��

��

��

��

��

��

��

9�

��

��

��

��

��

��

��

��

10�

��

��

��

��

��

��

��

��

11�

��

��

��

��

��

��

��

��

12V

�V

●V

●�

�V

V●

�●

V�

V13

��

��

��

��

��

��

��

14�

��

��

��

��

��

��

��

��

15�

��

��

��

��

��

��

�

16�

��

�V

��

��

��

��

�

17�

�●

��

��

��

��

��

��

�

18�

��

��

��

��

��

��

��

�

19�

��

�V

��

��

��

��

��

�

20�

�V

��

��

��

��

��

��

�

21�

��

��

��

��

��

��

��

22�

��

��

��

��

��

�V

��

�

23�

��

��

��

��

��

��

��

�

Note

:�

�80%

(orh

igher)

ratings

of“acceptable”

acrossstudies

��

70%(dow

nto

51%)

V�

50%(dow

nto

31%)

●�

30%(or

below)


Standards ratings by year. Ratings parti-tioned out by years reveal that: (a) the num-ber of studies conducted per year has steadilyincreased, and (b) the percentage of “accept-able” standards ratings within the currentstudy also steadily increased. Figure 4 displaysthese percentages across the ten-year exami-nation period.

Standards ratings by journal. Ratings of “ac-ceptable” across all journals fell below 80% infive areas: (a) description of participants (61%),(b) participation selection process (40%), (c)description of setting (71%), (d) implementa-tion fidelity (42%), and (e) controls for threats(78%). Approximately 16% of all ratings acrossall journals were in the “not acceptable” cate-gory (see Figure 5 for percentages of both rat-ings across all standards and journals).

Ratings by individual journals varied widelyacross the 23 standards. However, a pattern of“not acceptable” ratings can still be seen instandards 1 (description of participants), 2(participant selection process), and 12 (imple-mentation fidelity; all as discussed above), andall journals received standards ratings at 80%or above in a number of areas (from 48% to91% of the 23 standards). Table 5 displays“acceptable” standards ratings by individualjournal and percentage groupings.

Summary and Conclusions

The purpose of this review was to provide ananalysis of EBP within single-subject researchconducted with learners with ASD by using

Figure 4. Percentage of standards ratings by year. Above each set of bars is the number of studies identifiedwithin the corresponding year. Note that at the time of the current study, only one article wasidentified from 2009, and therefore no comparable examination of standards ratings could takeplace for that year.


the standards for evaluating this type of re-search methodology as provided by Horner etal. (2005). Across seven journals and ten years,studies meeting the predefined acceptancecriteria (N � 160) were found to have charac-teristics that: (a) were consistent with previousresearch (e.g., low rates of reported treatmentfidelity, as also found by Wheeler et al., 2006)and (b) extend previous work in this area(e.g., low rates of adequate reporting of par-ticipant characteristics).

Within the limitations of the current study’sscope and definitions, few IVs identifiedwithin the literature meet the NPDC’s defini-tion for EBP for learners with ASD (i.e., threedifferent investigators or research groupsmust have conducted five high quality single-subject design studies). Though authors arewell diversified across all articles within thecurrent study, the number of studies within IVcategories becomes a confound (e.g., 11 outof 17 IV categories had fewer than 5 studies

across the review period), as well as an unevendistribution across the geographic origins ofthe research (a point of comparison high-lighted within the Horner model). In addi-tion, examinations of standards ratings acrossIVs and journals reveal a high variability acrossmost EBP standards, as well as a negative pat-tern within several standards integrally tied toexternal validity (e.g., description of partici-pants, participant selection process, descrip-tion of setting, and implementation fidelity).This issue is further compounded by the lownumbers of participants typically found in sin-gle-subject research, a function of the historical,philosophical, and experimental origins of thistype of methodology as well as certain designlimitations that make it difficult (but not impos-sible) to include in a study more than, for in-stance, three participants. (Especially in the caseof ASD, the argument of poor availability ofparticipants is quickly becoming less valid.)These findings raise substantial questions re-

Figure 5. Standards ratings across all journals. The black line represents the total percentage of “notacceptable” ratings.


garding the adherence of this body of researchto EBP standards. However, it does appear thatadherence to the standards has been improvingover time. This is illustrated in the fact that thenumber of single-subject research studies con-ducted with learners with ASD has tended toincrease from year to year, yet poor compliancewith EBP standards has tended to decrease,though poor compliance still seems to stub-bornly persist within the aforementioned areas.

Implications for Practice

Variance exists regarding the quality of re-search, and therefore the stakes of using thefindings of research in daily practice arequite high for all involved. This creates animpetus for practitioners to become moresophisticated consumers of research. How-

ever, as in other industries such as the med-ical and airline industries, it should not bethe responsibility of the consumer of re-search to perform regular, intensive qualitycontrol of the product. Rather, it is the re-sponsibility of the consumer to know impor-tant indicators of a potentially inadequateproduct in order to better safeguard healthand well-being, ultimately through the exer-cise of one’s consumer rights (e.g., to shopelsewhere or file a complaint with the ap-propriate regulatory agency). This impliesthat in addition to increasing scrutiny at thelevel of the consumer, we must also: (a)increase scrutiny at the researcher and peer-review levels, and (b) vary the extent andtype of scrutiny at each of these levels inaccordance with the roles and responsibili-ties of those who work within them.

Table 5

Grouped standards ratings across journals.

Standard ETDD ETC FOCUS JABA JADD JPBI RDD

1 � � � ● � � V2 V � ● � V V3 � � � � � �

4 � � � � �

5 � � � � � � �

6 � � � � � � �

7 � � � � � � �

8 � � � � �

9 � � � � � � �

10 � � � � � �

11 � � � � � � �

12 ● � ● � V V13 � � � � � � �

14 � � � � � � �

15 � � � � � �

16 � � � V �

17 � � � � � �

18 � � � � �

19 � � � � � � �

20 � � � � � �

21 � � � � � �

22 � � � � � � �

23 � � � � � � �

Note:� � 80% (or higher) studies met the standard� � 70% (down to 51%)V � 50% (down to 31%)● � 30% (or below)


Implications for Research

Studies need to improve on the identificationand description of participants, especiallygiven the variability of characteristics associ-ated with ASD and the issue of utility/applica-bility of research findings for practitioners.Many studies within this review appeared tohave utilized samples of convenience (i.e., re-cruiting participants who were “handy” ratherthan searching out and targeting specific in-dividuals based on relevant criteria). It is notclear whether this appearance was accurate orsimply due to a failure to report procedure, asis the potential issue with other quality indica-tors such as treatment fidelity. (Duly noted arecommon constraints such as limitations onmanuscript length, as imposed by most printjournals.)

Studies also need to improve the descrip-tion of the settings in which they are con-ducted. In order for accurate replication ofthe IV to occur, relevant environmental fea-tures must be replicated. Complex environ-ments such as “classrooms” or “schools” haveto be, within reason, deconstructed for theconsumer so that adequate external validitycan be confirmed through (in part) appropri-ate, accurate application of the IV in settingswith features as identical as possible to theoriginal.

As in a growing number of other studies(e.g., Gresham, MacMillan, Frankenberger, &Bocian, 2000; McIntyre, Gresham, Di-Gennaro, & Reed, 2007; Wheeler et al., 2006),the current study found alarmingly low levelsof reporting procedures and data related totreatment fidelity. Studies need to improvethe reporting of both procedures and data inthis area, for one cannot reliably assert that afunctional relationship between the depen-dent variable and the IV has been establishedif one does not adequately demonstrate thatthe IV was implemented as designed.

There is work to do at all levels in increasingthe identification and use of EPB in the edu-cation and treatment of learners with ASD.Though some indicators examined over timeshow that we are getting incrementally closerto this goal, other indicators show the ongoing

need for refinement of how we identify anduse EBPs for and with this population. As neweducational tools and treatments becomeavailable to us, this process will be never-end-ing. Therefore, as those who are advocates forand positive intervention agents in the lives ofpeople with ASD, professionals at all levels areencouraged to embrace and apply this impor-tant concept.

References

Centers for Disease Control (2009). Prevalence ofautism spectrum disorders–Autism and develop-mental disabilities monitoring network, UnitedStates, 2006. Morbidity and Mortality Weekly Re-port. December 18, 2009; 58 (SS-10).

Gresham, F. M., MacMillan, D. L., Frankenberger,M. E., & Bocian, K. M. (2000). Treatment integ-rity in learning disabilities intervention research:Do we really know how treatments are imple-mented? Learning Disabilities Research and Practice,15, 198–205.

Horner, R. H., Carr, E. G., Halle, J., McGee, G.,Odom, S., Wolery, M. (2005). The use of single-subject research to identify evidence-based prac-tice in special education. Exceptional Children, 71,165–179.

McIntyre, L. L., Gresham, F. M., DiGennaro, F. D.,& Reed, D. D. (2007). Treatment integrity ofschool-based interventions with children in theJournal of Applied Behavior Analysis 1991–2005.Journal of Applied Behavior Analysis, 40, 659–672.

National Professional Development Center (NPDC)on Autism Spectrum Disorders (2009, January).Evidence-based practices for children and youth withautism spectrum disorders. Retrieved from http://www.fpg.unc.edu/�autismPDC/resources/resources_public_ebp.cfm

Odom, S. L., Brantlinger, E., Gersten, R., Horner,R. H., Thompson, B., & Harris, K. (2005). Re-search in special education: Scientific methodsand evidence-based practices. Exceptional Children,2, 137–148.

Simpson, R. L. (2005). Evidence-based practicesand students with autism spectrum disorders. Fo-cus on Autism and Other Developmental Disabilities,20, 140–149.

Wheeler, J. J., Baggett, B. A., Fox, J., & Blevins, L.(2006). Treatment integrity: A review of interven-tion studies conducted with children with autism.Focus on Autism and Other Developmental Disabilities,21, 45–54.


Documents

An Analysis of Evidence-Based Practices in the Education ...daddcec.org/Portals/0/CEC/Autism_Disabilities/Research/Publications/... · An Analysis of Evidence-Based Practices in the