AUTHOR McGrew, Kevin S.; And Others TITLE · AUTHOR McGrew, Kevin S.; And Others TITLE Inclusion of Students with Disabilities in National. ... wow. rho. docvment. ... national and

DOCUMENT RESUME

ED 347 769 BC 301 387

AUTHOR McGrew, Kevin S.; And OthersTITLE Inclusion of Students with Disabilities in National

and State Data Collection Programs. Technical Report2.

INSTITUTION National Associaticn of State Directors of SpecialEducation, Washington, D.C.; National Center onEducational Outcomes, Minneapolis, IN.; Saint CloudState Univ., MN.

SPONS AGENCY Special Education Programs (ED/OSERS), Washington,DC.

PUB DATE Mar 92CONTRACT H159C00004NOTE 33p.

AVAILABLE FROM National Center on Educational Outcomes (NCEO),Publications Office, 350 Elliott Hall, 75 E. RiverRd., Minrnapolis, MN 55455 ($10).

PUB TYPE Reports - Research/Technical (143)

EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS *Data Collection; *Disabilities; *Educational

Assessment; Educational Change; Educational Policy;*Educational Quality; Elementary Secondary Bducation;Evaluation Methods; National Surveys; ProgramDevelopment; State Surveys; *Student Evaluation

ABSTRACTThis report examines the extent to which individuals

with disabilities are involved in national and state data collectionprograms that play a pivotal role in reform efforts focusing onmeasurement of educational indicators. Thirty national datacollection programs were identified and 9 of the 30 are reviewed. Theextent to which students with disabilities are included in stateoutcomes assessment activities was evaluated through a survey ofstate directors of snecial eOucation. The study found that mostexisting national and state data collection programs exclude largeportions of the student population with disabilities. Exclusionappears most prevalent in data collection programs that requirestudents to complete surveys or tests independently. Data collectionprograms that use third party informants typically exclude fewindividuals. The use of different exclusion guidelines across datasets can cause problems in comparing results obtained from differentdata collection programs. The study concludes that the ability toextract useful national and state policy-relevant information on theoutcomes of students with disabilities from national and state datacollection programs is seriously hampered by the extensive exclusionof portions of this populati.on. Recommendations for improvement areoffered. (Twenty-nine references) (JDD)

Reproductions supplied by EDRS are the best that r7an be madefrom the original document.

*************************************************R*********************

1111001111MBIff OF EllUiCA11011Wee Echrcation0 Rostarch to/proomoms

TIONAL RESOURCES INFORMATIONCENTER wow

rho docvment Deer reprOgluted arem:erred from me Remo or orgenuilttOo

oextipolloagMgt*. changes tune been made to employe

reprOduchort aulth

Pomte Of revs of opentormiNeteci lofts 6Ctpao nol roecesearoy rerpreserrf citto

OERI posdion o pobcy

Technical Report 2

Inclusion of Students with Disabilities inNational and State

Data Collection Programs

Kevin S. McGrewSt. Cloud State University

Maitha L. Thurlow, James G. Shriner and Amy N. SpiegelUniversity of Minnesota

National Center on Educational OutcomesUNIVERSITY OF MINNESOTA

March, 1992

The National Center onEducational Outcomes (NCEO)was established in October, 1990to work with state departmentsof education, national policy-making groups, and others tofacilitate and enrich thedevf lopment and use ofindicators of educationaloutcomes for students withdisabilities. It is believed thatresponsible use of suchindicators will enable studentswith disabilities to achievebetter results from theireducational experiences. TheCenter represents a collaborativeeffort of the University ofMinnesota, the NationalAssociation of State Directors ofSpecial Education, and St. CloudState University.

The Center is supportedthrough a Cooperative Agreement(H159C00004) with the U.S.Department of Education, Officeof Special Education Programs.Opinions or points of view do notnecessarily represent those ofthe U.S. Department of Educationor Offices within it.

NCEO Core Staff:

Robert H. RruininksStanley L. DenoCheri GilmanKevin S. McGrewJames G. Shriner, State SurveyInterview CoordinatorMartha L. Thurlow.Assistant DirectorJames E. Ysseldyke, Director

NASOSE CoutributingStaff:

Eileen AhearnLinda Lewis. Associate11 I rec to r

Report Design:

Jill T. Jackson

41

Additional copiesof this report canbe obtained for$10.00 front:

NCEOPublicationsOffice350 Elliott Hall75 E, River RoadMinneapolis, MN55455

Table of Conten s

Page

Executive Summary

Introduction 1

National Education Data 2

Criteria for Exclusion 7

Rates of Exclusion 10Variability in Implementation of Exclusion Guidelines 12

How Does Exclusion Occur? 13

Points of Exclusion 14

State Education Data 14

Criteria for Exclusion 17

Testing Accommodations for Students with Disabilities 17

Exclusion Rates 18

Conclusions 18

The Nature and Context of Exclusion 18

When and How Exclusion Occurs 20Impact of Exclusion 21

Implications and Recommendations 21

References 25

i)

List of Tables

Page

Table 1: Data Sets Identified by the National Education Goals Panel as Sources of Informationfor Six National Goals 4

Table 2: Preliminary List of Targeted National Data Collection Programs 6

Table 3: Descriptions of Nine Data Sets 8

Table 4: Decision Makers and Guidelines Used in Inclusion Decisions for Students withDisabilities in Select National Data Collection Programs 9

Table 5: Inclusion Rates for Individuals with Disabilities in Select National Data CollectionPrograms 11

Table 6: Testing Accommodations Used by States. 19

Table 7: Estimated Participation Numbers and/or Rates of States 19

List of FiguresPage

Figure 1: NELS:88 Base Year Sample Selection/Exclusion Decision Making 15

Figure 2: Number of States with Formal or Written Rules n=34) Reporting Guidelines forParticipation in General Education Assessment 16

Inclusion of Students with Disabilities in National and StateData Collection Programs

ExecutiveCalls for reform in American education

during the past decade have resulted in raisedexpectations, attempts to develop unifrm and"world class" standards, and increasedemphasis on school accountability and themeasurement of educational outcomes. Reforminitiatives at both national and state levels arefocusing more frequently on outcomes andquantifiable data. The measurement ofeducational indicators is playing a centralrole in the current wave of educational reformas various groups seek to produce policy-relevant information on the educationalperformance and status of children in ournation's schools. With increasing frequency,the data needed to monitor and evaluate thesereform initiatives are being drawn fromnational and state data bases.

This report summarizes activities of theNational Center on Educational Outcomes(NCEO) that are directed at determining theextent to which individuals with disabilitiesare involved in national and state datacollection programs that are playing a pivotalrole in the current measurement-drivenreform movement. Inclusion of individualswith disabilities in the major national datacollection programs was evaluated byidentifying 30 different data collectionprograms that are receiving significantattention in the current reform initiatives.For this report, 9 of the 30 targeted nationaldata collection programs were reviewed. Theextent to which students with disabilities areincluded in state outcomes assessmentactivities was evaluated through a survey ofstate directors of special education. Togetherthese activities produced the following resultsand conclusions:

Most existing national and state datacollection programs exclude largeportions of the student population withdisabilities.

At the national level, it is estimated thatapproximately 40% to 50% of school-agestudents with disabilities are excludedfrom prominent national data collectionprograms.

Exclusion appears most prevalent in datacollection programs that requirestudents to complete surveys or testsindependently. Data collection programs

Summarythat are based solely on survey researchmethods typically exclude fewindividuals with disabilities when theyuse third party informants.

State-level data documenting the extentof exclusion of students withdisabilities is, with a few exceptions,largely unavailable at this time.

Exclusion of students with disabilitiesfrom national and state data collectionprograms occurs at many differentpoints, from the development ofassessment instruments, to the reportingof results.

Exclusion criteria typically areimplemented by local school personnelwho understandably are most concernedabout their immediate setting and whomay not appreciate the potentialusefulness of such information forschool improvement and the developmentof educational policy.

The use of different exclusionguidelines across data sets can causeproblems in comparing results obtainedfrom different data collection programs.Reasons typically given for exclusion ofstudents with disabilities range fromconcerns about providing properaccommodations (e.g.. in testadministration mode, in response mode,in flexible time or setting) to concernsabout the potential aversiveness of theassessment situation for the student.

A sizable portion of excluded studentsshould not have been excluded from datacollection programs, and could readilyparticipate (some with testingaccommodations, others without) in suchdata collection programs.

The ability to extract useful nationaland state policy-relevant information onthe outcomes of students withdisabilities from national and state datacollection programs is seriouslyhampered by the extensive exclusion ofportions of this population. Theexclusion of students with disabilitiesresults in significant problems inobtaining representative samples.

Executive

This, in turn, creates difficulties inestimating national and state levelstatistics (e.g.. dropout rates) and incompletini, accurate policy studies.

In conclusion, large numbers of studentswith disabilities are currently treated as"outliers" in our national and state datacollection programs. This categoricalexclusion of students with disabilitiesperpetuates the myth of inherent differences.Given the magnitude of federal and statesupport for educational programs for studentswith disabilities, support that reflects thevaluing of this population in our society, it istime that this implied value is matched by thecommitment of resources to address thenumerous political and technical hurdles thatmust be overcome in order for these studentsto participate more fully in our national andstate data collection programs. Current andfuture activities of the NCEO are being focusedin this direction.

A number of recommendations forincreasing the participation of students withdisabilities in national and state datacollection programs are offered. Theserecommendations focus on:

Developing broader and more uniformdefinitions of sample eligibility

Increasing adherence to inclusionguidelines, particularly the "if in doubt,include" component of inclusionguidelines

Developing sampling frames for datacollection programs that are moreinclusive

Routinely conducting follow-up studies ofineligible students as part of datacollection programs

Increasing the partial participation ofstudents with disabilities duringinstrument development

Researching and developing assessmentmodifications, accommodations, oralternatives that allow more students withdisabilities to participate in large scaleda:a collection programs

iv

Inclusion of Students with Disabilities in National and StateData Collection Programs

Assessment is a multimillion dollarenterprise in America today. Corporationsassess individuals to identify who should beemployed or promoted. The military assessesnew personnel to determine the types of jobsto assign to them. Schools assess children andyouth to ascertain the extent to which they areprogressing in school and to identify who mayhave unique educational needs. Federal andstate governments use information collectedby means of assessment to describe the statusof students and schools in the nation and ineach state. This information is used foraccountability purposes, and for settingeducational policy.

Within the past decade, concern has beenexpressed in reports about education inAmerica. Since the publication of k Nation atRisk (National Commission on Excellence inEducation, 1983), many reports have lamentedthe failure of our schools to adequatelyeducate students. While data were used todemonstrate education's lack of success (e.g.,Kirsch & fungeblut, 1986; NAEP, 1985;National Governors' Association, 1986; Officeof Educational Research and Improvement,1988), there was still dissatisfactionexpressed about the adequacy of availableeducational data.

During the 1980s, a wave of smtolegislation was passed on school report rerds.merit schools, and interstate achievementcomparisons (Kirst, 1990). To some extent, itwas believed that better educational datawould pressure schools to improve, which inturn would lead to better student performanceon tests, and even better preparation forstudents as they became members of theAmerican workforce and competitors in theinternational marketplace. The validity ofthis logic has been questioned by many (c.g..Darling-Hammond, 1991; Oakes, 1991).

Still, the need for data to describe thestatus of students is generally accepted. Thereare many who argue that the data collectionmethods need to be improved to reflectassessment of higher order thinking or toincorporate authentic assessment procedures,but most still would agree on the need for thedata. Even a cursory review of the currenteducational reform literature leads to theconclusion that there is a hunger for policy-relevant information on the performance of

students in our educatie,ial system.

In response to a legislative mandate(Hawkins-Stafford Education Amendments of1988: PL 100-297), the National Forum onEducational Statistics was formed to examinethe extent to which overlap or gaps existed indata collected by federal or state entities.This effort was initiated specifically toinfluence the production of a nationalcooperative statistics system of useful andcomparable data. In a report entitled A GuicICto Improving. the_ National_Etwcatios_DataSystem (NESAC, 1990), one of the Forum'scommittees presented 36 recommendations forimproving the national education data system.These recommendations covered the datadomains of (a) background/demographics, (b)education resources. (c) school processes, and(d) student outcomes.

Another related effort occurring about thesame time, and also sponsored by theHawkins-Stafford Education Amendments, wasthe formation of a Special Study Panel onEducational Indicators. The purpose of thisgroup was to identify educational indicatorsand organize them into logical groupings. Intheir report (Special Study Panel on EducationIndicators, 1991), entitled EXIIICatiQjl Counts.it was argued that the success of reform effortsdepends upon the development of a

"comprehensive education indicatorsinformation system capable of monitoring thehealth of the enterprise, identifying problems,and illuminating the road ahead" (p. 6).Further, in arguing that educationalindicators should define the educationalagenda rather that reflect an educationalagenda (i.e., "An indicator system organizedaround today's goals cannot respond totomorrow's"; p. 10), thc Panel proposed sixissue areas in which educational indicatorsare needed: (1) learner outcomes, (2) qualityof educational institutions, (3) readiness forschool, (4) societal support for learning, (5)education and economic productivity, and (6)

A third majo-r effort was the NationalEducation Goals Panel, formed to create amechanism to monitor the nation's and states'progress toward meeting the Bushadministration's six national education goals(e.g., school readiness, 90% high schoolgraduation rate, etc.). Six resource groups

National Education Data

were established to identify the kinds of dataneeded to track progress on each of the goals,what kinds of data were available, and toproduce annual reports containing datarelevant to each goal. In their first annualreport (National Education Goals Panel,1991b), the Panel indicated that it hadbrought together "the most recent quality dataavailable on how well the nation andindividual states are doing in achieving theNational Education Goals" (p. 191). However,it also noted that "while the report isrelatively comprehensive in some areas, thereare also many gaps" (p. 191). The gapsidentified in the repoft consist primarily oflists of indicators needed for each goal, suchas data on how prepared children are to learnwhen they enter school (physically,emotionally, socially, and intellectually),comparable state-level data on high schoolcompletion and dropout rates, and so on.

The NESAC report, Education Counts andthe Goals Report all highlighted the need forbetter data on students, particularly enspecific subsets of students. Studentsconsidered to be disadvantaged and studentsfrom specific cultures (e,g., Hispanic, AfricanAmerican, Native American) were specificallytargeted for special attention. Only theNESAC report, however, highlighted the needto disaggregate data for students withdisabilities who are served by the nation'sspecial education system. Students with awide array of disabilities fit within thispopulation, including those with learningdisabilities, emotional disabilities, andspeech and language impairments, those withsensory disabilities such as hearingimpairments and visual impairments, andthose with multiple and more severedisabilities, typically involving significantmental impairments. Given that over 4.5million school-age youngsters receive someform of special education services, servicesthat arc provided at significant xpense to oureducational system. it is imperative that weexamine how these students are performing.

There are several sources of informationthat can be used to evaluate the performance ofstudents with disabilities. First, our nat;onhas a wide array of data bases withinformation that is relevant to individualswith disabilities. These data bases cover suchareas as education, housing, employment, and

health. Individual states also maintain databases, usually within their Departments ofEducation. When the Council of Chief StateSchool Officers (CCSSO. 1990) examined stateeducation indicators, it again stressed theimportance of identifying and collecting datathat are technically, financially, andeducationally feasible, and of improving thecomparability of education data coflectedacross state:. In discussing the future, thereport stated:

"...he years ahead will be difficult forthose trying to develop better data,but the results of their efforts will becritical if we are to do a better job ofmonitoring our educational system.(p. 6)

The CCSSO report does not address the needfor data on specific subsets of students.

The purpose of this report is to examinenational and state data bases to determine theextent to which students with disabilities areincluded in these collection systems. We firstexamine national data bases, then shift to stateeducational data bases. These data collectionsystems are examined in terms of inclusionrates, criteria for exclusion of a student froma data base, and special accommodations thatare made to ensure inclusion of students. Thisis followed by a summary of the criticalconcerns that are generated as a result of theanalysis of the inclusion of students withdisabilities in national and state datacollectioe systems. Several recommendationsfor dealing with the exclusion of students withdisabilities in national and state datacollection programs conclude the report. Morecomprehensive solutions and guidelines willbe addressed in a subsequent report.


The United States has long recognized thevalue of large-scale federally funded studiesto assess student progress. Data collectionprograms such as the National Assessment ofEducational Progress (NAEP - the "Nation'sReport Card"), the National Longitudinal Study(NLS). High School and Beyond (HSB), and therecent National Education Longitudinal Study(NELS) are some of the more recognizableefforts.

Unfortunately, it appears from ouranalysis that the sampling plans for thesestudies have not allowed for adequate analysesof subgroups of students with disabilities, asituation that has resulted in students withdisabilities being considered "outliers" in OUTnational education data collection system(Allen, 1989). Until the recent funding by theOffice of Special Education Programs of theNational Longitudinal Transition Study ofSpecial Ed-ication Students (NLTS), there hasbeen relatively little national level policy-relevant knowledge produced about theeducational status and performance ofstudents with disabilities. Given themagnitude of federally-mandated educationalprograms for students with disabilities, "thelack of adequately designed national studiesof handicapped youth is particularlyalarming" (Allen, 1989. p. 469).

The first step in aur analysis of nationaldata sets was to identify a preliminary list of"target" data sets for critical review. Twocurrent national initiatives were consideredin selecting data bases for further study: (1)the national education goals, and (2) theeducational indicator movement.

National educational goals. Probablyno single force has served to spearhead thecurrent wave of education reform as have theactivities that surround the NationalEducation Goals Panel (NEGP) and the newnational education strategy of the Bushadministration Ajperic_11 2000. If studentswith disabilities are to be included in currentand future discussions of national educationgoals, it is important to assess the degree towhich this population is included in themeasurement strategies that drive thesereform initiatives. This is important becauseit may be, as suggested by the Special StudyPanel on Education Indicators (1991), that "webegin to value only what we can measure" (13.5). There is the related concern that we maybegin to value only who we can measure. "Tocategorically exclude students withdisabilities perpetuates the myth of inherentdifferences. It makes students with handicapsnon-students and perhaps non-people"(NASOSE, 1988, p. 10).

NEGP formed panels to developrecommendations for identifying potentialindicators and strategies for assessing

3


progress toward the six national goals. In itsreport entitled Measuring Progress_ Toward the

aiid MeaLurement ilrate,giea (NationalEducation Goals Panel, 1991a), the Panelfrequently recommended the use of indicatorsfrom such data sets as NELS and NAEP.Although many of the recommended data setsand indicators focus on context, input, andprocess indicators, some do include outcomeindicators. Listings of the data sets arepresented by goal in Table 1,

Educational Indicator movement. Itis clear that the current educational reforminitiatives have produced a flurry of activityfocused on the identification of educationalindicators to measure progress toward goals.The development of educational indicatorsystems has become a big "business" in theUnited States (Odden, 1990), with nearly allnational or state level groups and agenciesbecoming involved in these activities (Smith,1988). As a result, many data sets includedin the national education data system havebeen examined for possible indicators withwhich to monitor the effectiveness of reformactivities.

Among the major organizations that areexamining existing national data bases toidentify indicators to measure the progress ofchildren and youth are the Council of ChiefState School Officers, the National Governors'Association (NGA), and Joining Forces (acoalition of the American Public WelfareAssociation and the Council of Chief StateSchool Officers), to name but a few. Thesegroups have all turned to national data basesfor indicators to monitor progress during thecurrent wave of reform. Most have producedreports on their recommendations. Thesereports were reviewed to identify thosenational data collection programs that arereceiving significant attention (e.g., NAEP,NELS) in the educational indicator and reformliterature.

Summary. Based on a review of thenational goals activities and educationindicators movement, 30 national datacollection programs were identified. This listof national data sets is presented in Table 2.

Nine of the data sets in Table 2 have beenreviewed for this report. We sampled these

11

Table 1: Data Sets Identified by the National Education Goals Panel As PossibleSources of Information for Six National Goals

Data Source

Goal I: School ReadinessNationwide Food ConsumptionSurvey, Continuing Survey of FoodIntakes

Division of Vital Statistics

National Health Interview Survey

National Household EducationSurvey

Current Population Survey, SchoolEnrollment Supplement;Head Start Program InformationReport Questionnaire

Current Population Survey

High School & Beyond; NationalEducation Longitudinal Study

General Educational DevelopmentTesting ServiceCommon Core of Data

Sponsor

National Assessment of EducationalProgress (NAEP)

Advanced Placement Tests

lligh School Transcript Studies

International Evaluation ofEducational AchievementState Provided Data

Goal 4: Science and MathNational Assessment of EducationalProgress (NAEP)

International Evaluation ofEducational Achievement

International Assessment ofEducational Progress

National Survey of Science anJMathematics Education

Dept of Agriculture

Indicator

Dept of Health and Human Services

Nutritional status

Birthweight; prenatal care timing;prer.atal status

Dept of Health & Human Services, Children's access to health careNat'l Center for Health Statis,

Dept of Education, National Center Home activities and attitudesfor Education Statistics

Dept of Commerce, Census Bureau Preschool program participation

Dept of Education Preschool program quality

Dept of Commerce, Census Bureau Proportion of 19, 20. 24, 25 yr oldswith high school credential

Dept of Education, National Center I Drop out precursors; dropout rates;for Education Statistics

American Council on Education

dropouts who complete high school

Dropouts who complete high school

Dept of Education, National Center I State-by-state school completionfor Education Statistics data

Dept of Education, National Center Student achievementfor Education Statistics

The College Board Number of Advanced PlacewmentTests; scores earned

Dept of Education, National Center High school course enrollmentsfor Education Statistics

Dept of Education, National Center lnternat'l achievement comparisonsfor Education Statistics

State reports on studentachievement

Dept of Education, National Ce liter Instructional practkes; studentfor Education Statistics achievement, attitudes: teacher

characteristics

Dept of Education, National Centerfor Education Statistics

Dept of Education, National Centerfor Education Stat stics


Achievement; instructionalpractices; teacher characteristics

Achievement

Teacher and pri cipal attitudes

4

Table 1. (ctintinued)

Data Source

Schools and Staffing Survey

National Education LongitudinalSurvey

Longitudinal Study of A mrricanYouth

lotegrated Postsecondary EducationData System

Survey of Graduate Studerts andl'ost Doctorates'

Goal 5: Literacy an.National Assessment of EducationalProgress (NAEP)

VCCOMIEMMINMR10018018%.06,141014510AIVA101

Sponsor_

Dept of Edacation, National Centerfor Education Statistics


National Science Foundation

Dept of Education, National Centel-for Education Statistics

National Science Foundation

Special Populations' Literacy Skills

Armed Service Vocational AptitudeBattery

Workforce Participation Survey

National Household EducationSurvey (NIIES)

Integrated Postsecondary EducationData System

National Longitudinal Study; HighSchool & Beyond; NationalEducation Longitudinal Study

National Adult Literacy Survey

Dept of Education, National Centerfor Education Statisttcs

Dept of Labor

Dept of Defense

Dept of Labor




Dept of Education, National Centerfor Education Stat sties

Monitoring the Future

National Adolescent School HealthSurvey

National Crime Survey, SchoolCrime Supplement

National Education LongitudinalStudy

hist Response Survey System

Youth R isk Behavior SurveillanceSystem



Dept of Justice



Dept of Health & Human Services,Center for Disease Control

IndicatorInstructional practices and teacher

characteristics

Instructional practices and teacher

characteristics

Student and parent attitudes

Number of postsecondary graduates

Number of graduate studentenrollees

Literacy

Literacy

Vocational aptitudes

Skills preparation

Participation in adult learningprograms

Enrollment and graduation frompostsecondary training programs

Numkr of associate and bachelorsdegree recipients

Literacy

Drug use; crime victimization;attitude toward drugs

Drug use

Perceptions of safety in school;weapons in school

Student & teacher views of noise,disruption on learning

Crime victimization; student &teacher views of noise, disruptionon learning

Drug and tobacco use, weaponownership and use

13

Table 2: Preliminary List of 114 CEO Targeted National Data CollectionPrograms

"*Atettk*W~ANASOP.ON04%,*W..

DEggiumENLDE.Eaugillai

High School and Beyond

Transcript Study

Integrated Postsecondary Education DataSystem

International Assessment of EducationalProgress

National Adult Literacy Survey

National Assessment of EducationalProgress: 1988, 1990

National Assessment of EducationalProgress: Trial State Assessment

National Education Longitudinal Study

National Longitudinal Transition Study ofSpecial Education Students

National Household Education Survey

International Evaluation of EducationalAchievement

Young Adult Literacy Survey

Beginning Postsecondary Students

Baccalaureute and Beyond

PEPARTMENT OF COMMERCL

Curr"nt Population Survey

Survey of Income and ProgramParticipation

PEPARTMENT OF LABOR

Workforce Participation Survey

Workplace Literacy Assessment

DEEARIAIEN.LDLEISLICE

National Crime Survey

DEPARTAZNT _OF HEALTH ANDHUNAN SERVICES

National Health Interview

National Health and NutritionExamination Survey

National Survey of Personal HealthPractices and Consequences

National Survey of Family Growth

National Adolescent School HealthSurvey

Youth Risk Behavior SurveillanceSystem

National Household Survey of DrugA buse

Monitoring the Future

NATIONAL SCIENCE FOUNDATION

Survey of Graduate Students and PostDoctorates

Longitudinal Study of American Youth

AMERICAN COUNCIL QF FeDUCATION

General Education DevelopmentTesting

THE COLLEGE BOARD

Advanced Placemem Tests


data sets from several sponsoring agencies sothat we could examine variability amongsponsoring agencies as well as among datasets. The specific data sets (and theirsponsoring agencies) that were selected forreview in this report are:

National Adult Literacy Survey(Department of Education) NALS

National Assessment of EducationalProgress: 1988 (Department of Education)NAEP:88

N tional Assessment of EducationalProg ess: 1990 (Department of Education)

NAEP:90

National Assessment of Edu:ationalProgress: Trial State Assessment Program(Department of Education) -- NAEP:TrialState

1111National Education Longitudinal Study of1988 (Department of Education) NELS:88

1111 Current Population Survey, MarchSupplement (Departmert of Commerce)CPS

National Health Interview Survey(Department of Health and HumanServices) NHIS

National Health and NutritionExamination Survey. Natie lel HealthEpidemiological Follow-up Study(Department of Health and HumanServices) NHANES, NHEFS

IlLongitudinal Study of American Youth(National Science Foundation) LSAY

These data sets arc described in Table 3.

For each of the nine national datacollection programs, all relevant reports andmanuals (methodological and technical) wereobtained from the sponsoring agency andreviewed as part of a detailed "'disabilitysensitivity review" (McGrew, Spiegel, Thurlow,Ysseldyke, Bruininks, Deno, & Shriner.1991a). The relevant documentation for eachtargeted data set was -eviewed to extract thefollowing information:

I. Descriptive ittioxmation Title,collection cycle, research design.sponsor. contact, and general purpose ofthe data zollection program.

2. 5 oprce and method of data collectionInformation on who (student, parent,teacher, administrator) provides thedata, and how the data are collected(questionnaire, test, records review,e:e.)

3. Samole_ tutalyj is - Description ofsampling design, with particularattention to the use of disability relatedexclusionary procedures, operationaldefinition of disability categories, andthe disability characteristics of thefinal sample.

4. Description of indicetors Descriptionof the general domains of variablescontained in the data set.

For the current report, the information ofinterest was obtained as part of the "sampleanalysis" component. I s componentprovided information on: (a) the disability-related inclusion and exclusion guidelinesused in the data collection program. (b) whomakes the inclusion and exclusion decisionsduring data collection, and (c) the rates ofinclusion or exclusion of students withdisabilities in the final sample.

Criteria for Exclusion

The information presented in Table 4 is asummary of the different disability-relatedexclusion guidelines and procedures used bythe nine selected national data collectionprograms. A number of conclusions can bedrawn from the information presented in Table4.

First, there is considerable variability inthe extent to which exclusion guidelines areoperationalized and reported in the differentdata collection programs. However, some ofthese differences are understandable based onthe different assessment methods that areused. In data collection programs that onlyrequire the completion of an interviewprotocol (viz., NH'S. NHEFS, CPS), exclusion of

Table 3: Descriptions of Nine Data Sets

National Adult Literacy Survey (Department of Education) -- NALS

A nationally representative cross-sectional study designed to collect information on the types and levelsof literacy skills adults living in the United States possess and how these skills are distributed acrossmajor subgroups. This study is assessing the prose, document, and quantitative literacy of approximately15,(XX) adults (16 to 64 years of age) in 1992.

National Assessment of Educational Progress: 1988 (Department of Education) -- NAEP:88

National Assessment of Educational Progress: 1990 (Department of Education) NAEP:90

National Assessment of Educational Progress: Trial State Assessment Program (Department ofEducation) -- NAEP:Trial State

NAEP is a nationally representative cross-sectional study designed to monitor the knowledge, skills,understanding, and attitudes of the nation's children and youth. This data collection program began in1969 and currently assesses different curriculum areas (e.g., reading, writing, mathematics, science,citizenship, U.S. history, geography, social studies, art, music, literature, career and occupationaldevelopment) in grades 4, 8, and 12 every two years. Two years (1988 and 1990) as well as the voluntar}state program started in 1990 (the State Trial) were reviewed for this report. The State Trial providedstate-level mathematics data for eighth graders for 40 participating jurisdictions.

National Education Longitudinal Study of 1988 (Department of Education) NELS:88

A nationally representative longitudinal study designed to assess the baseline experiences of eighth gradestudents and to relate these experiences to current academic achievement and to later achievement inschool ard life. The 1988 base year data collection program gathered data in a variety of areas such aswork status, values, school characteristics, school atmosphere, school work, school performance,guidance, special programs, after-school supervision, involvement with community, after schoolactivities, educational and occupational life goals, and financial assistance. Follow-up assessments arebeing completed every two years from 1990 to 1996.

Current Population Survey, March Supplement (Department of Commerce) -- CPS

A nationally representative cross-sectional study designed to collect information on the employmentsituation and demographic status of the complete U.S. population (birth through adulthood). The MarchSupplement is specifically designed to gather data on work experience, income, noncash benefits, andpopulation migration. Data collection in this program has been conducted annually since the 1940s.

National Health Interview Survey (Department of Health and Human Services) NHIS

A nationally representative cross-sectional study designed to provide information on the health of thecivilian noninstitutionalized U.S. population (birth through adulthood). This survey has been completedannually since 1957. While the same basic demographic and health-related information is collected eachyear, additional information on special health topics (c.g., AIDS, aging, etc.) may becovered in any onesurvey.

National Health and Nutrition Examination Survey, National Health Epidemiological Follow-upStudy (Department of Health and Human Services) NHANES, NHEFS

A nationally representative longitudinal study designed to (a) provide information on the prevalence ofhealth conditions and risk factors, (h) monitor changes over time in health, functional status, andutilization of hospitals, and (c) track the incidence of various medical conditions in the U.S. population(birth through adulthood). The base year data are drawn from the National Health and NutritionExamination Survey 1 (NHANES 1), with the follow-ups in 1982-84, 1986, 1987, and 1991.

Longitudinal Study of American Youth (National Science Foundation) -- LSAY

A nationally representative longitudinal study of seventh and tenth graders designed to assess studentattitudes toward science and mathematics as areas of study and possible career choices. Base year datacollection started in 1987, with annual follow-ups.

. . ........

8

I G

Table 4: Decision Makers and Guidelines Used in Inclusion Decisions forStudents with Disabilities in Select National Data Collection Programs

mwtoconmet.",nzorecYmmit.tt.'4...1Mw(s^...o.w.;m:,tcov.kM,*.x wzmmmed.mi......kavo.loxwoo,."1.V:temetv

Ag en cy/ Prog ram Who Disability-Related Exclusion Guidelines

National Assessment ofEducational Progress, 1988(NAEP:88)

National Assessment ofEducational Progress, 1990(NAEP:90)

NAEP Trial StateAssessment Program, 1990

National EducationLongitudinal Study of 1988(NELS:88), Base Year

National Adult LiteracySurvey, 1992 (NALS:92)Base Year

(NCES)School Students on sampling roster who were deemed to be untestablestaff and unable to participate meaningfully in the assessment.

Disability related ineligibility categories used were:Mild retardation (educable) Functional disability

Schoolstaff

Schoolstaff

Schoolstaff

Inter-viewer

National Hea th Interview NASurvey, 1989 (NHIS:89)

Students on sampling roster who were deemed to be unassess-able and unable to participate meaningfully in the assessment.Disability-related ineligibility categories used were:

Student is in special education with an IEP and ismainstreamed less than 50% of the time in academicsubjects and is judged incapable of participatingmeaningfully in the assessment.Student is in special education with an IEP andthe IEP team Or equivalent group has determined that thestudent is incapable of participating meaningfully in the

assessment.

(Same as NAEP:90 above)

Students on sampling roster designated eligible ifdetermination was made that student was capable of completingthe survey instruments, and designated as ineligible if judgedthat student would be incapable of doing so. Disability relatedineligibility categories used were:

Severe mental disability Physical disability

Individuals on sampling roster who are unable to complete thebackground questionnaire or literacy exercises due to:

Language problem Physical or mental disabilityReading difficulty

Nat'l Health & Nutrition Ex- NAamination Survey I (NHANESI) Epidemiologic FollowupStudy, 1986 (NHEFS:86)

(NCHS)No disability-related exclusion guidelines; individuals withdisabilities are not systematically excluded. Informationregarding individuals with disabilities who arc included onthe sampling roster is collected through adult proxies.

(Same as NHIS:89 described above)

DEPARTMENT OF COMMERCE, Census Bureau)Current Population Survey(CPS)

NA (Same as NHIS:89 described above)

NATIONAL SCIENCE FOI,INDATI,ONLongitudinal Study of School No formal disability-relatedAmerican Youth, 1987 stall. Only mention of exclusion of(LSAY:87), Base Year

exclusion guidelinesstudents on sampling

in regards to students who declined or refused to

reported.rosters wasparticipate.

'' . . .. .. . ..

9

17


individuals wilt' disabilities is not asignificant issue, since third party informantsor proxies can provide the necessaryinformation for individuals who havedisabilities that preclude their owncommunication with interviewers. In the caseof these data collection programs, no specificexclusion guidelines are specified, nor arethey needed.

This contrasts with those data collectionprograms that require the selected individualsto respond themselves to actual test items orsurvey instruments (viz.. NAEP, NAEP TrialState Assessment, NELS, NALS, LSAY). In allof these data collection programs there is anexpressed concern for individuals who areunable to participate meaningfully in theassessment due to some form of disability.The rationales advanced for exclusiontypically revolve around a concern for notsubmitting individuals to a very stressful andoften futile attempt at data collection, asituation that may also produce results ofquestionable quality.

Although there is generally a commonground in the reasons stated for exclusion andfor who makes the exclusion decisions(usually local school staff), little in the way ofcommon ground is found across the sampleddata collection programs in operationalguidelines. At one extreme is the inability tofind any formal documentation of disability-related exclusion guidelines for LSAY:87.Inspection of the relevant LSAY:87 documents,including a search of the files by the currentstaff managing the program, failed to find anyspecific mention of disability-I-elatedexclusion guidelines for LSAY:87. In contrast,NAEP:88 and NELS:88 were similar in thatthey each suggested two possible categories(although the categories were different) forexclusion based on some form of disability;NAEP:88 used "mild retardation (educable)"and "functional disability" while NELS:88used "severe mental disabilities" and"significant physical disabilities." Reflectingchanges in methodology, the more recentNAEP:90 and NAEP Trial State AssessmentProgram of 1990 use exclusion categories tiedto whether students are on an activeIndividualized Educational Plan (IEP), and theextent to which they participate inmainstream education.

.........

1 0

-------

NALS:92 plans to use yet a different set ofexclusion guidelines, probably because it doesnot focus on a captive school population withreadily available special educationinformation on students. Individuals unableto complete the background questionnaire orliteracy exercises due to language, visualimpairment, physical or mental disability, orreading difficulties will be excluded. Thesedeterminations apparently will be made by thetrained interviewers who have contact with thesampled individuals. Ironically, individualswith reading difficulties, who by definitionare of greatest concern in the area of literacy,will be excluded because of their difficulty inrea4".ng.

In summary, national data collectionprograms appear to vary markedly in theextent and type of disability-relatedexclusion guidelines used during datacollection. Some of this variability can beattributed to differences in data collectionmethods, with those not requiring respondentsto complete survey instruments independentlyspecifying no exclusion rules. However, evenamong those data collection programs that usesimilar data collection methods (e.g., tests),significant variability is noted in exclusionguidelines for individuals with disabilities.

Rates of Exclusion

Information on estimated exclusion ratesfor the nine data sets is presented in Table 5.Similar to the observed dichotomy in thenature of disability-related exclusionguidelines, the data collection programsappear either to exclude relatively fewindividuals with disabilities (e.g., NHIS,NHEFS, CPS) or to exclude approximately 1/3to 1/2 of school age students with disabilities.Again, this dichotomy can be attributed to thedifference in methods of data collection notedpreviously.

With the exception of NALS:92, for whichfigures are not yet available, and LSAY:87, forwhich no figures were reported, all datacollection programs listed in Table 5 thatrequire direct testing of students (NAEP,NAEP Trial State Assessment Program, NELS)exclude approximately 1/3 to 1/2 of all schoolage students with disabilities. As noted inTable 5, these estimates are based only on that

Table 5: Inclusion Rates for Individuals with Disabilities in Select NationalData Collection Programs

Ag en cy / Frog ram Estimated Percent with Disabilities Excludeda

PFPASTMENT _OF EDUCATION (NCES)National Assessment of EducationalProgtzss, 1988 (NAEP:88)

NAEP Trial State Assessment Program,1990 (NAEP: Trial State)

National Assessment of EducationalProgress, 1990 (NAEP:90)

National Education Longitudinal Studyof 1988 (NELS:88)

National Adult Literacy Survey, 1992(NALS:92) Base Year

Average exclusion rate for total sample: approx. 5.7%, whichincludes students v, ith limited English proficiency (LEP). Ifassume 113 of excluded students were LEO). 3.8% of theoriginally sampled students were excluded due to disability-related guidelines. Approximately 40% of stpdents with

Average exclusion rate across samples: approximately 4.4% forstudents with IEPs. Average of 52.7% of students with lEriselected for the __Aiunples suludect.

(Assume same as NAEP Trial State Assessment Program,1990)d

Exclusion rate for total sample: approximately 5.4%, whichincludes 1.9% students with limited English proficiency(LEP). 3.4% of the originally sampled students excluded dueto disability-related guidelines. Approximately 36% ofstudentl_ with disabilities ex.cluded froni t sample.c

Information not yet available since survey will rmt be imple-mented until 1992.

DEFE1RTMENT OF HEALTH AND HUMAN SERVICES (NCHS)National Health Interview Survey, 1989(NHIS:89)

Nat'l Health & Nutrition ExaminationSurvey I (NHANES I) EpidemiologicFollowup Study, 19136 (NHEFS:86)

DEPARTMENT OF CONIMERCECurrent Population Survey (CPS)

NAIMAL_af,Longitudinal Study of American Youth,1987 (1.SAY:87), Base Year

No figures reported since no disability-related guidelines areused. Exclusion is probably negligible in amount.

(Same as NHIS:89 described above)

Census Bureau)(Same as NHIS:8) described above)

No figures reported since no disability-related guidelines aredocumented. Any informal/formal exclusion guidelines mostlikely resulted in exclusion rates similar to those of programswith similar data collection instruments e. . NAEP NELS

27010WeIrMk4iCaMA:=4:0C , VSSMWSMIMZ:2;::;s:).ms.Stargiard.ORW .~$3Maaata'araita Estimates only reflect exclusion of student population attending regular schools (does not reflect students with disabilities inseparate facilities).

b1988 NAEP technical report only provides exclusion figures as a total and does not report a breakdown by different exclusioncategories. The use of the 1/3 figure for LEP students is drawn from the rate of LEP exclusion reported for thc NELS:88 studywhich occurred at the same time.

cEstimate of total percent of indiv iduals with disabilities excluded calculated by comparing reported exclusion percent (for disability-related reasons) in total sample with average percent of student population with disabilities (not including separate facilities). Sinceaverage values reported during rment years indicate approximately 10% of the student population can be classified as having adisability, and since approximately 7% of this population receives services through separate facilities, a value of 9.3% was used inthese calculations. This represents the percentage of students with disabilities who were included in the sampling frame.

dFinal technical report for NAEP:90 not available at time of this report. The procedures used were like those employed in the1990 NAEP Trial State Assessment, suggesting that exclusion rates are probably comparable. However, analysis by Spencer(1991) suggests that NAEP:90 may have excluded more than the 1990 NAEP Trial State Assessment. based on a comparison ofpercent of tested students in the samples with IEPs.

11


portion of the school age population withdisabilities that is receiving special educationservices through regular education, resourceroom, or separate special education classes.These figures do not reflect the additionalexclusion of students with disabilities thatoccurs as the result of data collectionprograms starting with a sampling universethat excludes separate special educationfacilities (e.g.. residential, homebound,hospital, separate school settings). It isestimated that approximately 7% of the schoolage population receives special educationservices in such separate environments. Innumbers, this is approximately 315.000students. The exclusion of these separateenvironments from the sampling universe isbased on the rationale that essentially allstudents served in these settings have scrim'sor multiple disabilities that preclude theirmeaningful participation in data collection,and that all such students would probablyqualify under a data collection program'sexclusion guidelines. However, such a blanketassumption is flawed. Many students inresidential settings are capable ofparticipating in standardized testingprograms, and often do so on a regular basis(NASDSE, 1988).

Regardless of the exact proportion ofstudents in separate facilities who couldparticipate in data collection programs (evenif this number were quite small relative to theentire sample size), the important point is thata significant portion of the student populationwith disabilities (i.e., all students in separatefacilities) is ignored in much of thc nationaleducation data collection system as currentlydesigned. The undercoverage of thetotal school age populatiLn of studentswith disabilities Is estimated to bebetween 40% and 50%.

Variability in Implementation ofExclusion Guidelines

Although certain of the data collectionprograms listed in Table 4 specify exclusionguidelines, a review of the relevant manualsfrom these programs suggests that little, ifany, information is typically collected on thedegree to which the guidelines areimplemented consistently. Recently.information from two of the data collection

MINIOMINIIIINEEMEM

1 2

programs (viz.. NAEP Trial State Assessment;NELS) suggests that there may be significantvariability in application of exclusionguidelines within data collection programs.

As reported in Table 4, approximately52.7% of all selected students with IEPs wereexcluded from the 1990 NAEP Trial StateAssessment data collection. State-by-stateexclusion figures (Houser, personalcommunication, February, 1991) foundexclusion rates for students with IEPs to be aslow as approximately 33% (Minnesota) and ashigh as approximately 71% (Arkansas), withan even higher rate of 87% for thc District ofColumbia. Given that tte same designprocedures and exclusion guidelines wereused in both the national and state 1990 NAEPassessments, one could conclude thatconsiderable variability in t heimplementation of disability-relatedexclusion guidelines appears to occur in boththe national and state NAEP assessments.

The follow-up study being completed on asample of students determined to be ineligiblefor the NELS base year data collection(NELS:88) is providing important insights intothe exclusion process (Ingels, 1991). In thecase of NELS, as well as the other datacollection programs listed in Table 1 that haveexclusion guidelines, the application of theexclusion guidelines and all related decisionsare made by local school staff. In thedirections to local staff during NELS base yeardata collection. schools were asked to applythe exclusion guidelines on an indi-vidualbasis, and not to exclude studentscategorically. In the case of uncertainty,school personnel were asked to include thestudent.

Preliminary results from the NELSIneligible Study indicated that despite thesedirections, schools often resorted to thecategorical exclusion of students (Ingels,1991). Evidence for this categorical exclusionwas observed in thc exclusion of all studentswithin a specific category on some of theschool sampling rosters, In addition, ifschools had been making individual exclusiondecisions about students with disabilities andincluding students when in doubt, one wouldexpect to find a number of students who wouldexperience difficulty during the actualtesting. In contrast, exceptionally high

National Education Dataviimmanwavaimmampaum

assisted telephone interviews, estimates thatapproximately 8% of households wereexcluded because they did not have telephones(West, Hausken, Chanchei, & Collins, 1991).Households without telephones tend to differin substantive ways from those withtelephones, usually in terms of including ahigher percentage of low-income households.Other persons excluded when telephonesurveys are used include persons who are deafor who use telecommunication devices (Kiecolt& Nathan, 1985).

completion rates (i.e.. exceeding 99%) werereported for all tests. This leads to theconclusion that "the screening out of studentswas too effective in that one would expect moreborderline cases had schools taken with fullseriousness the injunction 'when in doubt,include'" (Inge ls, 1991, p. 11).

Further evidence reflecting arbitraryexclusion during some of the data collection isfound in preliminary results front the NELSIneligible Study during the first follow-upsurvey (S. J. Inge ls, personal communication,June 25, 1991). These results indicated thatof the approximately 94% of the students inthe NELS base year (1988) ineligible samplethct were studied during the first follow-up(1990), over half (58.5%) were reclassified aseligible. More importantly, approximately94% of those reclassified as eligible were ableto successfully complete the data collectioninstruments. Unfortunately, the preliminaryinformation available at the time this reportwas written did not provide for adisaggregation of the ineligible follow-upresults for just those excluded due todisability-related guidelines (e.g., the dataalso include students excluded due to limitedEnglish proficiency). However, one probablycan assume from all of the preliminaryevidence reported from the NELS IneligibleStudy that significant numbers of studentswho were deemed "ineligible" during baseyear data collection due to a specificdisability should not have been excluded andapparently can successfully participate inthese types of data collection activities. Mostlikely these results also generalize to otherdata collection programs (e.g., NAEP) sincemany are directed by the same agencies andmany rely on local implementation ofexclusion rules.

How Does Exclusion Occur?

The exclusion of certain segments of thepopulation from large scale data collectionprograms occurs in a variety of ways. Forexample, the increased use of computer-assisted telephone interviewing in place offa.7e-to-face interviewing has resulted in theomission of households without telephones.The recent National Household EducationSurvey (NHES; National Center for EducationStatistics, 1991). which uses computer-

1 3

As mentioned previously, a substantialproportion of students with disabilities areregularly excluded from national educationsurveys. Special schools for individuals withdisabilities, which are attended by as many as7% of all students with disabilities, are notincluded in most sampling frames. Inaddition, students with disabilities attendingregular schools are frequently excluded. Forexample, in HSB three sampling constraintshave led to the non-inclusion of a segment ofstudents in the data base (Owings & Stocking,1985). First, only students who were involvedin high school programs leading to graduationand a diploma were included. Consequently,those students in nondcgree programs (such asstudents in special education who wereworking toward a certificate of completion)were not included in the sample. Second,those students who could not read andcomplete the questionnaire on their own wereexcluded, eliminating, among others, studentswho were blind or who had difficulty usingpencils. Third, those students judged byteachers to be "at risk" for experiencingdiscomfort during the testing situation alsowere excluded. Consequently, a substantialproportion of students with mental, emotional,and/or physical disabilities were excludedfrom the HSB sample.

Exclusion also probably occurs for anumber of reasons that have not been verifiedin the literature. As noted above, studentsmay be excluded if it is perceived that theywould experience discomfort during thetesting situation. Some have suggested that anextension of this may be that students who donot want to take a test (translated to the testsituation would cause discomfort) maysimilarly be excluded. Beyond this, it alsohas been suggested that administrators mayexclude students if it is perceived that the

State Education Data

students' test scores would lower theperformance levels reported for a school or adistrict.

Points of Exclusion

A detailed review of the sample selectionand exclusion procedures used for NELS:88illustrates those study design points wherestudents with disabilities typically areexcluded from national data collectionprograms. NELS:88 is only used forillustrative purposes, and is not being singledout as "the- example to illustrateexclusionary practices. In fact, the NELS:88survey is probably the most prominentexample of recent attempts to address theissue of exclusion of students withdisabilities from survey samples. Even thoughexclusion rules were used during thecollection of the base year data for NELS:88,extensive follow-up studies are beingcompleted to reassess the continued status ofexcluded students, and where appropriate, toadd such students back into the study atsubsequent follow-ups (Inge Is, 1991). Thesample exclusion process of NELS:88 shown inFigure 1 is used only as an example toidentify those points where students withdisabilities may be systematically excludedfrom large scale sampling plans.

The five shaded boxes in Figure 1 aredecision-making points that frequently resultin the exclusion of students with disabilitiesfrom large scale assessment programs. First, asignificant proportion of the studentpopulation with disabilities is excluded whenseparate schools for individuals withdisabilities are excluded from the samplingframe. Second, the use of school rosters basedon grade placement also results in theexclusion of any students in ungraded specialeducation classes. Additional exclusionoccurs through the application of formalexclusion criteria at t*o other decision-making points (pre-roster ineligibilitydecision; sample updating). These result instudents being excluded if physical or mentaldisabilities are present that, in the judgmentof school staff, preclude independentcompletion of the survey instruments. Thefinal exclusion point may occur at the time ofthe administration of the survey if studentsare observed to experience difficulty

independently completing the instruments.

Of the possible exclusion pointshighlighted in Figure 1, the last three arethose for which data typically are gathered forreporting exclusion rates. These reportedexclusion rates only represent the percent ofstudents with disabilities who are excludedfrom the selected sample, and are most likelyunderrepresentations of the total number ofstudents with disabilities who are excluded.This number is already lowered when entireschools are excluded and when students are inungraded programs. The information conveyedby Figure 1 suggests that a sizable portion ofthe total student population with disabilitiesis excluded from most national data collectionprograms.


Just as federal initiatives have engenderedinterest in assessing student progress, so havestate-level initiatives. Increasing numbers ofstates are involved in collecting outcomes datafrom schools in order to determine whetherstate goals are being met.

Data relevant to state exclusion policiesare available from a survey of states conductedby the National Center on EducationalOutcomes (NCEO, 1992; Shriner, Bruininks,Deno, McGrew, Thurlow & Ysseldyke, 1991).In the Spring of 1991, state directors ofspecial education or their designeesresponded to a telephone survey on a varietyof issues related to the assessment ofoutcomes. Seven survey questions addressedthe issue of exclusion of students withdisabilities from state-level outcomesassessments. These questions were:

Do any students with disabilities takepart in tests given to general educationstudents that are collected and/orreported at the state level?

What decision rules or guidelines areused to determine which studentsparticipate and which are excluded?

Who makes the decision to include orexclude a student with disabilitiesregarding the outcomes assessment ingeneral education?

1 4

2

Figurg NELS:118 Base Year Sample Selection/Exclusion Decision Making

All public/ pnvate schoolsw ith 8th graciers

Exclude

Exclude separate spec ed. BIA,area vocational, and overseasschools

Include Select sample of 'regular' public/private schools 1,057 of1,655 selected participated

'$.0601:rogit-O4bi ut:40

Exclude

Students in 'ungraded programsmay have been excluded from bthgrade rosters

floater deciS ion ^toiltert miewad byWW1 Coartinater acding to7 inebility codes

EXCLUDED BASE YEAR STUDENTStn.10,853, 5 35 % of potent' a!school sample)

-physical disabihty tn.840, 0 41 g)

-mental disabihty (n-6,182, 3 04 %)

language problem ln-3,861, 1 90

GELS YO ineligible f all('w -Back Study)

Exclude

Include

FINAL BASE 1'EAR 8TH GRADESTUDENT SAMPLE

-Student Questionnaire tn-24.599)-Student Test fr123.7011-Parent Questionnaire (n22,6511-Teacher Ouestionnaire tn-23,188)-School Admin Quest ionnaire(ri-1,0351

fn-completed ca es)4.11===xmon.

Select initial sample

ALS 90 F rst Follow-Up Survey )

thadate sample tpost-rosterineligiblity decision) gstopte lletreviewed Jointly by surveyorgantrlitlori retweeentative andschool coordinator.

iInclude

Survey session - additional 0051-roster inelivittl !Sty deCisions maderegarding stwents unaple tocomplete teStsilSurvErtS

1 5

igur e 2: Number of States with Formal or Written Rules (n=34) ReportingGuidelines for Participation in General Education Assessments*

ModificationsThreaten

AdverseStudent R

IEP NotAddressingConcern

InvalidatTest Score

Degree ofImpaiiment

LimitedProficiency

Level ofService

CoursesMainstream

Percent TimeGenera!

StudentCharacle

LocalDelerrnin

Number of States

18

* Source Shnner, 1 ruininks, Deno, McGrow, Thurlow, & Ysseldyke, 1991.

16


Are these decision rules formal orinformal?

Would you share a copy of theseguidelines with the Center?

Can students with disabilities whoparticipate in the assessments beidentified in the data set?

About how many students withdisabilities do you believe participateeach year in the assessments conductedfor students without disabilities?

Criteria for Exclusion

The NCEO survey revealed that 49 of the50 states reported that students withdisabilities participate in the assessment ofstudent achievement. Within these states, 34states indicated that they have formal orwritten decision rules for the exclusion ofstudents with disabilities. The decision rulesgenerally are guidelines for districts and localschools. Eight types of guidelines encompassthose used by most St tiles to determinewhether a student will participate in thegeneral education assessment. Theseguidelines, which do overlap, include:

Locally-determined rules forparticipationStudent specific characteristicsPercent of time in general educationclassesCourses for which student ismainstreamedLevel of service receivedDocumented circumstances thatinvalidate test scoresIEP does not address test contentDegree of student impairment

The number of states using each of theseguidelines is displayed in Figure 2.

When states were asked to report on whomakes the decision about inclusion of studentswith disabilities in state testing, 37 of the 49states with students participating in sometype of outcomes assessment identified localIEP teams. Other states indicated a principal(n = 8), parents (n = 9), or some other unique

I 7

decision maker (n = 15); five states indicatedthat inclusion decisions were made by thestate education agency. Of the 34 states withformal or written guidelines that addressedthe issue of inclusion of students withdisabilities in general education achievementtesting, 27 (79%) identified the IEP team asthe decision maker.

The consistency with which exclusionguidelines are implemented is another issue.While considerable effort has been expendedin states to develop guidelines to assure theproper inclusion of students with disabilities,some State Directors of Special Education hadsignificant doubts about how guidelines wereoperationalized and implemented. The specialeducation units of most state agencies do notdirectly monitor inclusion decisions since itis typically the general education unit thatconducts state assessments. As such, theinclusion/exclusion decision is influenced byfactors not directly related to the student. Forexample, in states where scores determinevarious types of funding allocations, studentswith disabilities may be categoricallyexcluded in order to increase the probabilityof higher or lower school sco:es, depending onwhich would better serve the purposes of aparticular school.

Even when students with disabilitiesparticipate in state assessments, there is noguarantee that their data are included instate-level analyses and/or reports.Inconsistent handling of data after it leavesthe local school site is quite common.Although not formally asked in the survey ofstates, respondents frequently expressedconcern that many students are beingexcluded post hoc. In other words, their testprotocols are simply discounted or discarded.Students are exposed to assessments thatproduce possibly valuable information, yetthese data from the assessments are not usedto enhance the students' education in any way.

Testing Accommodations for Studentswith Disabilities

Thc NCEO state survey results alsorevealed that one of the most importantdeterminants of whether a student with adisability actually participated in state-leveltesting was whether the state or the local

Conclusions

school allowed special testing accommodationsfor the student. Forty-two states reported thatthey provide one or more forms of testingaccommodations for students with disabilities.However, only 28 states had written guidelinesabout these accommodations. The nature of theaccommodations identified by the 42 states isshown in Table 6. The most frequentaccommodations were alternative presentationmodes (e.g.. Braille test, oral reading of test tostudent, etc.). Flexible testing times andflexible settings also were identified by alarge proportion of the states.

Exclusion Rates

The rates at which students withdisabilities arc excluded from stateassessment programs seems to be largelyunknown at this time. When State Directors ofSpecial Education or their designees wereasked to specify the number or percent ofstudents with disabilities who participated intheir state-level general education testingprograms, many states (n = 34; 68%) did nothave an idea of the extent of participation.Some respondents were uncertain whetherstudents with disabilities could be identifiedin their data sets. The 19 states that didreport participation rates are shown in Table7. Of the 13 states giving percentages, 5 statesindicated that more than 90% of their studentswith disabilities participate in state testing, 5indicated that between 50% and 90%participate, and 3 states indicated that lessthan 50% of their students with disabilitiesparticipate in testing. Of the three states withthe loNest percentages, two had less than 5%of students with disabilities participating instate-level testing.

Conclusions

Calls for reform in American educationduring the past decade have resulted in raisedexpectations, attempts to develop uniform and"world class" standards, and increasedemphasis on school accountability and themeasurement of educational outcomes. Reforminitiatives, both at the state and national level,are increasingly focusing on outcomes andquantifiable data. The measurement ofeducational indicators is playing a centralrole in the current wave of educational reform

as various groups seek to produce policy-relevant information on the educationalperformance and status of children in ournation's schools. With increasing frequency,the data needed to monitor and evaluate thesereform initiatives are being drawn fromnational and state data bases.

The Nature and Context of Exclusion

As currently designed. most of the existingnational and state data collection programsexclude large portions of the studentpopulation with disabilities. At the nationallevel, it is estimated that approximately 40% to50% of all school-age students with disabilitiesare excluded from the most prominent nationaldata collection programs (e.g., NAEP, NELS)that are playing a critical role in the evaluationof the current reform initiatives. Indicative ofthe low status that the school-age disabilitypopulation has in data collection programs isthe finding that state-level data documentingthe extent of exclusion is, with few exceptions,largely unavailable at this time.

It is clear that the ability to extract usefulnational and state policy-relevant informationon the outcomes of students with disabilities isseriously hampered by the extensive exclusionof portions of this population. Given thecurrent assessment technology, some exclusionof students with unique needs isunderstandable and cannot be avoided. Uniquetesting accommodations cannot address alldisability-related problems in large-scalenational and state assessment programs.

Our review of national and state datacollection programs suggests, however, that asizable portion of excluded students should nothave been excluded, and could readilyparticipate (some with testine accommodations;others without) in such data collectionprograms. Contributing to this problem is thesignificant variability in the types of exclusioncriteria used in national and state datacollection systems. As indicated by the NELSIneligible Study and the state NAEP results,even when exclusion criteria are in place,implementation of the criteria are variable, andmay be affected by local decisions, whichsometimes are driven by a concern forproducing high scores in assessment results.In general, little effort has been made to

abILA: Testing Accommodations Used by States

Conclus onssystematically study ways to increase the rateof participation of students with disabilitiesin many, if not most, large-scale assessmentprograms.

When and How Exclusion Occurs

The exclusion of students with disabilitiesfrom national and state data collectionprograms occurs at a number of differentstages: (a) during the development ofassessment instruments, (b) when the data arecollected, and (c) during the analysis,interpretation, and reporting of the results.

Our review of the methodology reports ofmost national assessment programs revealedlittle description of how students withdisabilities are included in the itemdevelopment phase of assessment instruments.Furthermore, most sampling plans for nationaldata collection programs systematicallyexclude certain segments of the school-agepopulation with disabilities (e.g., students inresidential, homebound, hospital, or separateschool settings). The logic in assuming thatall of these students are incapable ofparticipating in standardized testingprograms is flawed, since many of thesestudents actually are involved in standardizedtesting on a regular basis (NASDSE, 1988).This situation is analogous to the exclusion ofstudents with disabilities in the developmentand standardization of many norm-referencedtests (Fuchs, Fuchs, Benowitz, & Barringer,1987).

Commercial tests commonly used in stateassessments (e.g., Stanford Achievement Test)typically do not include students withdisabilities in the development stages of thetest, or if they do, they do not report thisinformation. Although the inclusion ofrepresentative samples of students withdisabilities is not necessarily required for thecalculation of accurate norms in wellstandardized instruments, the inclusion ofsuch individuals in test development iscritical from the perspective of learning whatitems or test administration procedures mayneed to be dropped or modified in order toaccommodate the unique testing needs of thispopulation (Daniels, 1989).

The exclusion of students with disabilities

2 0

during the actual collection of data isprobably the most obvious type of exclusion.The stated and unstated reasons for not givingmany students with disabilities theopportunity to participat- in the assessmentare varied. Reasons range from a concern overthe inability to provide properaccommodations (e.g., in test administrationmode, in response mode, in flexible time orsetting) to a concern about the potentialaversiveness of the assessment situation forthe student. The most common state-levelcriteria used for these exclusion decisions arebased on (a) local rules about participation,(b) information regarding student specificcharacteristics, or (c) amount of time thestudent spends in general education classes.Although the exclusion criteria for nationaldata collection programs may differ somewhatfrom those used by state programs, in bothcases, exclusion criteria are typicallyimplemented by local school personnel. Asnoted by Ingels (1991). the specification offormal exclusion criteria by no meansguarantees accurate implementation. Localimplementation variability appears to becommonplace. It is possible that many localschool staff who make these decisions, whounderstandably are most concerned abouttheir immediate setting, do not appreciate the"bigger picture" of national and state testing,and therefore do not incorporate into theirdecisions the usefulness of such informationfor school improvement and the dev?lopment ofeducational policy.

Exclusion in the analysis andinterpretation phases of assessments occurswhen students who were included in theassessments are excluded from the dataanalysis. This type of exclusion is common instate-level assessments, possibly due to a

desire to maintain higher school, district, orstate level results in the current era of highstakes testing. From the results of the NCEOstate survey, it seems that students withdisabilities present many challenges to stateagencies that try to incl, de them in stateassessments. Often, an easy solution is toremove their data from those of the "regularstudents." States are aware that data onstudents with disabilities might be retrievedand analyzed, but have not directed consistentefforts toward this task. In addition, fewspecial education sections in statedepartments of education use the data

separately from general education for specialeducation related policy decisions. Even inthose states that collect data that can bedisaggregated separately for students withdisabilities, state directors of specialeducation sometimes are unaware that thiscapability exists.

Impact of Exclusion

The exclusion of large numbers of studentswith disabilities from national and stateassessment programs raises questions aboutthe inferences that can be drawn from theresults of these data collection programs. Thisexclusion may result in significant problemsin estimation of national and state levelstatistics, the ability to obtain representativesamples, and the ability to complete accuratepolicy studies (Inge ls, 1991).

Not only does the treatment of studentswith disabilities as outliers in national andstate data collection programs mke itdifficult to produce accurate national andstate statistical estimates for this population.it also raises questions about bias beingpresent in most national and state educationstatistical estimates that are reported. Forexample, dropout rates for certain subgroups(e.g., students with disabilities, students withlimited English proficiency) of the populationmay be significantly higher than the generalpopulation. If large portions of thesesubgroups are excluded from the calculationof national and state dropout statisticalestimates, the estimates may be biased. Whenthe exclusion occurs for subgroups thattypically have the most difficulty in learningand functioning within the schools (e.g.,students with disabilities), the resultingestimates most likely will paint a moreoptimistic picture of educational progress.especially for those subgroups that are notadequately represented. Ironically, in suchsituations it is those who are excluded who aretypically targeted for intervention programsand are those groups most often of greatestpolicy interest (Inge Is, 1991).

Other problems can also result from theexclusion of students with disabilities fromnational and state data sets, particularly whendata collection programs use differentexclusion procedures and criteria. Samples

Igkications and Recommendationsthat result from the implementation ofdifferent exclusion guidelines can causeproblems with sample comparability (McGrew,Spiegel, Thur low, Ysseldyke, Bruininks, Deno,& Shriner, 1991b). That is, differences insample composition can make it difficult tomake comparisons across data bases. Inaddition, exclusion procedures may make itdifficult to study small groups (e.g.,subgroups of individuals with disabilities)because of the resulting small sample sizes,and more critically, because of the significantproblems this introduces into the estimationof appropriate standard errors of the estimatefor statistics for these small subgroups(McGrew, Spiegel, Thurlow, Ysseldyke,Bruininks, Deno, & Shriner, 1991b).

2 1

Finally, the treatment of most studentswith disabilities as "outliers" in our nationaland state data collection programs is a concernfrom an equity and philosophical perspective.The categorical exclusion of studf_nts withdisabilities "perpetuates the myth of inherentdifferences. It makes students with handicapsnon-students and perhaps non-people"(NASDSE, 1988, p. 10). Given the magnitude offederal and state support for educationalprograms for students with disabilities,support that reflects the valuing of thispopulation in our society, it is time that thisimplied value is matched by the commitmentof resources to address the nurrwrous politicaland technical hurdles that must be overcomein order for these students to more full,participate in our national and state datacollection programs. Current and futureactivities of the NCEO are being focused inthis direction.

Implicationsand Recommendations

Identifying and describing a problem ismuch easier to accomplish than is developingplans to effectively address the problem. Inthis report, we have demonstrated that largenumbers of students with disabilities areexcluded from national and state assessmentprograms. Multiple political and technicalissues underlie this exclu;ion. Addre.;singthose issues is more difficult than simplydescribing them. While we believe thatfurther study and empirical research areneeded before it is possible to produce a

implications and Recommendationscomprehensive set of guidelines for includingall students in assessment programs, it isposs ible to m ak e some in i ti alrecommendations.

Seven recommendations are presentedhere. This list is not exhaustive and iscontinually being modified and expanded aspart of ongoing NCEO activities. More detailedreports addressing these and additionalrecommendations will be produced in thefuture.

Develop broader and more uniformdel.:Miens of sample eligibility.

The need for a broad and inclusivedefinition of sample eligibility that would beuniform across data sets has been pointed outby others (e.g., Ingels, 1991). We found thatsignificant variability exists between andwithin national and state data collectionprograms in their definitions and criteria forsample eligibility. This makes comparisons ofresults across data sets difficult. With acommon system of disability definitions andcategories, it would be possible not only tomake comparisons across data sets but also todevelop larger data sets constructed byintegrating information from different datasets.

Increase adherence to inclusionguidelines.

Significant exclusion occurs in datacollection programs as a result ofinappropriate implementation of inclusion andexclusion guidelines. This is evident in theobserved variability in the inclusion ofstudents with disabilities among states in theNAEP Trial State Assessment Program, forexample. It is also evident in the findingsfrom the NELS Ineligible Follow-up Study,which revealed frequent categorical exclusionof students with disabilities (often involvingstudents who were capable of participating inthe data collection). It is clear that the "if indoubt, include" component of most inclusionguidelines is not consistently followed. Theuse of procedures that involve contactingschool staff who have the most directknowledge about students with disabilities(e.g., special education teachers instead ofschool office personnel) may help local staffbe more consistent in their inclusiondecisions (Ingels, 1991). Mechanisms need tobe developed to insure greater adherence to

2 2

uniform implementation of the "if in doubt,include" component of inclusion guidelines innational and state data collection programs.Possible mechanisms include monitoringsystems and incentive programs.

Develop sampling frames that are moreinclusive.

Many students with disabilities are neverconsidered for inclusion in national and statedata collection programs due to samplingplans that routinely exclude them. Forexample, separate schools for students withdisabilities are almost always excluded fromlarge scale assessment sampling plans. Yet,many of these schools serve students who areable to complete standardized assessmentinstruments. The use of grade-based schoolrosters can also result in the exclusion ofstudents ungraded special educationprograms. Efforts need to be directed towardinsuring the inclusion of these excludedportions of the student population in datacollection sampling plans.

Include follow-up studies ofineligible students as a standardcomponent of data collection programs.

Greater efforts arc needed to study thecharacteristics of students with disabilitieswho are excluded from data collectionprograms in order to accurately estimate theeffect of this exclusion on importantstatistical estimates and to allow re-entry intothe sample when the data collection effort islongitudinal. The NELS Ineligible Follow-upStudy is an excellent example of attempts toaddress more systematically the issue ofexclusion of individuals from data collectionprograms. This study has demonstrated thevalue of following samples of excludedstudents to (a) delf.mine whether eligibilitystatus changes with time so that initiallyexcluded students can be folded back into thesample, and (b) gather information on thecharacteristics of excluded students todetermine whether adjustments orqualifications need to be mad , in statisticalestimates drawn from the complete sample.

Increase partial participation in datacollection programs.

Given the current assessment technology,it is unreasonable to expect that all studentswith all forms of disabilities will be able toparticipate in the same way in all components

Implica ions and Recommendations

of national and state data collection programs.Yet, there are opportunities for the collectionof partial ir.ormation that are overlooked inmany data collection programs. Typically.data collection programs that require theindep-ndent completion of tests or surveys(e.g., NAEP, NELS) exclude a student from allcomponents of the data collection program ifthe student is unable to participate in thosecomponents requiring independentcompletion. Yet, these data collectionprograms often collect additional informationfrom third party proxies (e.g., teachers orparents) on important outcome variables.Students with disabilities should be includedin data collection components that rely onproxies or other indirect methods (e.g.,administrative record reviews). Attentionneeds to be directed to insuring that thesedata collection opportunities are not lost forstudents with disabilities.

Include students with disabilities Ininstrument development.

The deliberate inclusion of students withdisabilities, particularly those for whom anassessment may be relevant and appropriate,during the initial stages of instrumentdevelopment, is encouraged. T..rough suchinvolvement, those responsible for thedevelopment of assessment instruments candiscover items, questions, tasks, orprocedures that may need to be eliminated ormodified in order to allow these students toparticipate in the data collection activities.

Develop assessment modifications,accommodations,and alternatives.

Probably the most critical barrier to theinclusion of more students with disabilities innational and state data collection programs isour limited knowledge of what modificationsor accommodations can be made to currentassessment tools. Research and developmentactivities must focus on investigating theextent to which modifications and/oraccommodations can be made to instrumentsused in large-scale data collection programs,without destroying the basic psychometricproperties of the instruments. A variety ofmodifications and accommodations should bestudied, including flexible scheduling.flexible settings, revised test formats, revisedtest directions, revised response formats.and/or the use of aides (NASDSE, 1988). Inaddition, research and development efforts

2 3

need to be focused on the use of, ordevelopment of, alternative assessmentmethodologies or techniques (e.g., computerassisted testing; use of item responsetechnology) that would allow greater numbersof students with disabilities to participatemore fully in large-scale assessmentprograms.

ReferencesAllen, T. E. (Ed.). (1989). Handicapped youthand the outcomes of their schooling Resentresearch advances in the United States [SpecialIssue]. InjujudjuzaLlournaLof Educational&maul. 13-

CCSSO. (1990). tate education indicators;1 98 9. Washington. DC: Author.

Daniels, M. H. (1989. April). Issues andAdvances jn norming ability tests. Paperpresented the annual convention of theNational Association of School Psychologists,Boston.

Darling-Hammond, L. (19 91). Measuringschools is not the same as improving them. InVoices from the field: 30 expert opinions ouAmeuica 2009. the Bujh aAministratioustrategy to "jeinvent" America's schools.Washington. DC: Youth and America's Future,William T. Grant Foundation.

Fuchs, D., Fuchs, L. S., Benowitz, S., &Barringer, K. (1987). Norm-referenced tests:Are they valid for use with handicappedstudents? E xx,ep tie n al Children. 5A, 263-272.

Ingels, S. I. (1991. April). The problem ofexcluded baseline stuclents_in_a_school-based

drQmout estimates and accommodatingcligthilimshAns,S_ALSLAims. Paper presentedat the annual meeting of the AmericanEducational Research Association. Chicago.

Kiecolt, K. J., & Nathan, L. E. (1985).S_econdary analysis of sqrvgy data. BeverlyHills. CA: Sage.

Kirsch, I. S., & Jungeblut, A. (1986).Literagy; Profjles of America's Young adults.Princeton, NJ: Educational Testing Service.

Kirst, M. W. (1990). Accountability;Implications far state and local policy makers.Policy perspectives. Washington. DC:Information Services office of EducationalResearch and Improvement, U. S. Department ofEducation.

McGrew, K. S., Spiegel, A., Thurlow, M. L.,Ysseldyke, J. E., Bruininks, R. H., Deno, S. L.,& Shriner, I. G. (1991a). Plan for identifyingancLanalyzing_national and liale data kasgs by

2 5

the National Cerise; on 4clucittional Outcomes.Minneapolis, MN: National Center onEducational Outcomes, University ofMinnesota.

McGrew, K.S., Spiegel, A., Thurlow, M.L.,Ysseldyke, J.E., Bruininks, R. H., Deno, S.L., &Shriner, LG. (1991b). &ecekdary dateanattais: A reYiew of imaisr conceptional,MA as tu e tite n t . and t cc hake &j issumMinneapolis. MN: National Center onEducational Outcomes, University ofMinnesota.

NASDSE. (1988). NAEP testingjor statessuniadigia;lijuumaragissLisLiadjuicuLgfhandickgsed students. Washington, DC:National Association of State Directors ofSpecial Education.

National Center for Education Statistics.(1991). Netional HouseholdEducatiot Survcy(NHES). Washington, DC: U.S. Department ofEducation.

NAEP (Nationial Assessment of EducationalProgress). (1985). The reading report card.Princeton, NI: Educational Testing Service.

National Commission on Excellence inEducation. (1983). IVnation at risk: Theimperalive for educational _reform.Washington, DC: U. S. Government PrintingOffice.

National Education Goals Panel. (1991a).Measttrin4 progress tomard ths nationaleducation goa1s1 Potuitial indicators andmeasuiernent strategies (DiscussionDocument). Washington, DC: Author.

National Education Goals Panel. (1991b). Thenational_ educationitoala report 1991: Building

nation of learners. Washington, DC: U.S.Government Printing Office.

National Governors' Association. (1986).Time fin results: The Lovernors' 1991 report.Washington, DC: Author.

NCEO. (1992). state special educationoutcomes 1991: A report on state activities inthe asies_sment of educational outomes felsiudents with disxbilities. Minneapolis, MN:National Center on Educational Outcomes.

./ 4,

References4121111111111111111111111111

NESAC (National Education Statistics AgendaCommittee of the National Forum on EducationStatistics). (1990). Inintovinit our nationaleilloaiszLILALILIIALCIIIEAlLiacmiLlar_ALL1211.Washington, DC: Author.

Oakes, J. (1991). The many-sided dilemmasof testing. In Voices from the field: 30_expertgyiniona iati America 2000. tht ilushadministration stratel,y lo "reinvent"America's schools, Washington, DC: Youthand America's Future, William T. GrantFoundation.

Odden, A. (1990). Educational indicators inthe United States: The need for analysis.Educatjonal Researcher, 1...2 (5), 24-28.

Office of Educational Research andImprovement. (1988). Youth indicators 198$1Trends_ in the well-being of Americail yojith.Washington, DC: U. S. Government PrintingOffice.

Owings. J.A., & Stocking. C. (1985). HighSchool and Beyond: Characteristics oChighschool students svho_identify rhgmitelves ashandicapped. Washington, DC: U.S.Department of Education.

Shriner, J. G., Bruininks, R. H., Deno, S. L.,McGrew, K. S., Thurlow, M. L., & Ysseldyke. J.E. (1991). State practices in the assgssmenrof outcomes for slit_clen_tit with disabilities(Technical Report). Minneapolis: NationalCenter on Educational Outcomes.

Smith, M. S. (1988). Educational indicators.phi Delia Kaopan. 0_2(7), 487-491.

Special Study Panel on Education Indicators.(1991). E ucfftion counts: An indiciktors_xstem to monitor the ruttioit's educatiorulhealth. Washington, DC: National Center forEducation Statistics.

Spencer, B.D. (1991). ElialhilikyJczclusionissues in Ole 1990 Trial StAtt. sscssmenL.Evanston. IL: Methodology Research Center,NORC, Northwestern University.

West, J., Hauskcn, E. G, Chandler, K. A., &Coffins, M. (1991). Experiences in child careand early childhood programs of first andsecond graders prior to entering first grade:Findings from the 1991 National Household

2 6

Education Survey. In Statistics in Brief.Washington, DC: Office of Education Researchand Improvement, U.S. Department ofEducation.

Documents

AUTHOR McGrew, Kevin S.; And Others TITLE · AUTHOR McGrew, Kevin S.; And Others TITLE Inclusion of Students with Disabilities in National. ... wow. rho. docvment. ... national and