Jamal Abedi University of California, Davis/CRESST Validity, Effectiveness and Feasibility of Accommodations for English Language Learners With Disabilities

Jamal AbediJamal AbediUniversity of California, Davis/CRESSTUniversity of California, Davis/CRESST

Validity, Effectiveness and Validity, Effectiveness and Feasibility of Feasibility of

Accommodations for Accommodations for English Language English Language

Learners With Disabilities Learners With Disabilities

(ELLWD)(ELLWD)

Accommodations Accommodations for ELLWDfor ELLWD

The process of provision of accommodations The process of provision of accommodations to ELLWD is more complex than the process to ELLWD is more complex than the process used for either SD or ELL students.used for either SD or ELL students.Care must be taken to choose Care must be taken to choose accommodations that are appropriate for this accommodations that are appropriate for this particular subgroup of students. particular subgroup of students. ELLWDs need accommodation addressing ELLWDs need accommodation addressing both their language needs and their both their language needs and their disabilities.disabilities.

Why Should English Why Should English Language Learners Be Language Learners Be

Accommodated?Accommodated?Their possible English-language Their possible English-language deficiency may interfere with their deficiency may interfere with their content knowledge performance.content knowledge performance.Assessment tools may be Assessment tools may be culturally and linguistically biased culturally and linguistically biased for these students.for these students.Linguistic complexity of the Linguistic complexity of the assessment tools may be a source assessment tools may be a source of measurement error.of measurement error.Language factors may be a source Language factors may be a source of construct irrelevant variance.of construct irrelevant variance.

Why Should Students Why Should Students With Disabilities be With Disabilities be

AccommodatedAccommodated??

Their disabilities Their disabilities put them at put them at disadvantage.disadvantage.

Accommodations Accommodations must be provided must be provided to level the to level the playing field.playing field.

Subgroup Reading Math Language Spelling

ELL Status

ELL

Mean 26.3 34.6 32.3 28.5

SD 15.2 15.2 16.6 16.7

N 62,273 64,153 62,559 64,359

Non-ELL

Mean 51.7 52.0 55.2 51.6

SD 19.5 20.7 20.9 20.0

N 244,847 245,838 243,199 246,818

SES

Low SES

Mean 34.3 38.1 38.9 36.3

SD 18.9 17.1 19.8 20.0

N 92,302 94,054 92,221 94,505

Higher SES

Mean 48.2 49.4 51.7 47.6

SD 21.8 21.6 22.6 22.0

N 307,931 310,684 306,176 312,321

Site 2 Grade 7 SAT 9 Subsection Scores

Reading Math Math Calculation

Math Analytical

Non-ELL/SWD

Mean 45.63 49.30 49.09 48.75

SD 21.10 20.47 20.78 19.61

N 9217 91.18 9846 92.50

ELL only

Mean 20.26 36.00 39.20 33.86

SD 16.39 18.48 21.25 16.88

N 692 687 696 699

SWD only

Mean 18.86 27.82 28.42 29.10

SD 19.70 14.10 15.76 15.14

N 872 843 883 873

ELL/SWD

Mean 9.78 21.37 22.75 22.87

SD 11.50 10.75 12.94 12.06

N 93 92 97 94

Site 4 Grade 8 Descriptive Statistics for the SAT 9 Test Scores by Strands

ReadingReading Science Science Math Math MM SDSD MM SDSD MM SDSD

Grade 10Grade 10SWD onlySWD only 16.416.4 12.712.7 25.525.5 13.313.3 22.522.5 11.711.7LEP onlyLEP only 24.024.0 16.416.4 32.932.9 15.315.3 36.836.8 16.016.0LEP & SWDLEP & SWD 16.316.3 11.211.2 24.824.8 9.3 9.3 23.623.6 9.8 9.8Non-LEP/SWD 38.0Non-LEP/SWD 38.0 16.016.0 42.642.6 17.217.2 39.639.6 16.916.9All studentsAll students 36.036.0 16.916.9 41.341.3 17.517.5 38.538.5 17.017.0

Grade 11Grade 11SWD OnlySWD Only 14.914.9 13.213.2 21.521.5 12.312.3 24.324.3 13.213.2LEP OnlyLEP Only 22.522.5 16.116.1 28.428.4 14.414.4 45.545.5 18.218.2LEP & SWDLEP & SWD 15.515.5 12.712.7 26.126.1 20.120.1 25.125.1 13.013.0Non-LEP/SWD 38.4Non-LEP/SWD 38.4 18.318.3 39.639.6 18.818.8 45.245.2 21.121.1All StudentsAll Students 36.236.2 19.019.0 38.238.2 18.918.9 44.044.0 21.221.2

Normal Curve Equivalent Means & Standard Deviations for Students in Grades 10 and 11, Site

3 School District

Accommodations for Accommodations for ELLs and SWDsELLs and SWDs

Can the same accommodations Can the same accommodations used for students with used for students with disabilities be used for ELLs?disabilities be used for ELLs?Can the same accommodations Can the same accommodations used for ELLs be used for used for ELLs be used for students with disabilities?students with disabilities?

How Are We Doing in How Are We Doing in Practice Nationally?Practice Nationally?

• Are the states and districts across the Are the states and districts across the nation cognizant of this important nation cognizant of this important principle of using accommodations that principle of using accommodations that are appropriate for a particular are appropriate for a particular subgroup?subgroup?

• Are there any objective national criteria Are there any objective national criteria to help states to select appropriate to help states to select appropriate accommodations for ELLWD students? accommodations for ELLWD students?

• Or, is the assignment of Or, is the assignment of accommodations to these students based accommodations to these students based on temporary and subjective decisions?on temporary and subjective decisions?

SD LEP Accommodation

l l 1. Extension of allotted time l l 2. Use of multiple shortened test periods l l 3. Simplification of directions l l 4. Reading of questions (Math and Science only) l 5. Translation of words and phrases on the spot l l 6. Decoding of words upon request l 7. Use of calculator l l 8. Gestures and nonverbal expression l 9. Allow students to mark answers in test booklet l 10. Allow students to point to response l 11. Allow student to answer orally l 12. Allow student to respond on audio tape l 13. Use of typewriter or personal computer l l 14. Use of graphic organizers and art work l 15. Use of large-print or magnification device l l 16. Testing in separate room l l 17. Small group setting l l 18. Use of study carrel

Accommodation in Stanford 9 Testing Used by a District

SY 2000-2001 Accommodations Designated for ELLs Cited in

States’ Policies

There are 73 accommodations listed:

N: Not Related

R: Remotely Related

M: Moderately Related

H: Highly Related

From: Rivera (2003) State assessment policies for English language learners. Presented at the 2003 Large-Scale Assessment Conference

N 1. Test time increased

N 2. Breaks provided

N 3. Test schedule extended

N 4. Subtests flexibly scheduled

N 5. Test administered at time of day most beneficial to test taker

N = not related; R = remotely related; M = moderately related; H = highly related

I. Timing/Scheduling (N = 5)


States’ Policies

N 1. Test individually administered

N 2. Test administered in small group

N 3. Test administered in location with minimal distraction

N 4. Test administered in familiar room

N 5. Test taker in separate location (or carrel)

N 6. Test administered in ESL/Bilingual classroom

N 7. Individual administration provided outside school (home, hospital, institution, etc.)

N 8. Test taker provided preferential seating

N 9. Increased or decreased opportunity for movement provided

II. Setting (N = 17)

N 10. Teacher faces test taker

N 11. Special/appropriate lighting provided

N 12. Adaptive or special furniture provided

N 13. Adaptive pencils provided

N 14. Adaptive keyboards provided

N 15. Person familiar with test taker administers test

N 16. ESL/bilingual teacher administers test

N 17. Additional one-to-one support provided during test administration in general education classroom (e.g. instructional assistant, special test administrator, LEP staff, etc.)

SY 2000-2001 Accommodations Designated for ELLs Cited in States’

Policies


R 1. Directions repeated in English

R 2. Directions read aloud

R 3. Audio-taped directions provided in English

N 4. Key words or phrases highlighted

M 5. Directions simplified

M 6. Audio-taped directions provided in native language

M 7. Directions translated into native language

N 8. Cues provided to help test taker remain on task

M 9. Directions explained/clarified in English

III. Presentation (N = 32)

M 10. Directions explained/clarified in native language

M 11. Both oral and written directions in English provided

M 12. Both oral and written directions in native language provided

M 13. Test items read aloud in English

H 14. Test items read aloud in simplified/sheltered English

N 15. Audio-taped test items provided in English

H 16. Test items read aloud in native language

H 17. Audio-taped test items provided in native language


Policies


N 18. Assistive listening devices, amplifications, noise buffers, appropriate acoustics provided

N 19. Key words and phrases in test highlighted

H 20. Words on test clarified (e.g. words defined, explained)

H 21. Bilingual word lists, customized dictionaries (word-to-word translations) provided

N 22. Enlarged print, magnifying equipment, Braille provided

N 23. Memory aids, fact charts, list of formulas and/or research sheets provided

N 24. Templates, masks or markers provided

III. Presentation (N = 32)

N 25. Cues (e.g. arrows and stop signs) provided on answer form

N 26. Acetate shield for page provided

N 27. Colored stickers or highlighters for visual cues provided

R 28. Augmentive communication systems or strategies provided (e.g. letter boards, picture communication devices, voice output systems, electronic devices)

H 29. Simplified/sheltered English version of test provided

H 30. Side-by-side bilingual versions of test provided

H 31. Translated version of the test provided

N 32. Test interpreted for the deaf or hearing impaired/use of sign language provided


Policies


N 1. Test taker marks answers in test booklet

N 2. Test administrator transfers test-taker’s answers

N 3. Test taker’s transferred responses checked for accurate marking

N 4. Copying assistance provided between drafts

N 5. Test taker types or uses a machine to respond (e.g. typewriter/word processor/computer)

N 6. Test taker indicates answers by pointing or other method

N 7. Papers secured to work area with tape/magnets

N 8. Mounting systems, slant boards, easels provided to change position of paper, alter test taker’s position

N. 9. Physical assistance provided

IV. Response (N = 17)

N 10. Enlarged answer sheets provided

R 11. Alternative writing systems provided (including portable writing devices, computers and voice-activated technology)

R 12. Test taker verifies understanding of directions

R 13. Test taker dictates or uses a scribe to respond in English

N 14. Test taker responds on audio tape in English

H 15. Oral response in native language translated into English

H 16. Written response in native language translated into English

H 17. Spelling assistance, spelling dictionaries, spell/grammar checker provided


Policies


N 1. Out-of-level testing provided

N 2. Special test preparation provided

V. Other (N = 2)

Preliminary Findings: State Assessment Policies for English Language Learners, SY 2000-2001

GW/CEEE, Large-Scale Assessment Conference 2003

[email protected]


States’ Policies


There are 73 Accommodations Listed

47 or 64% are not related

7 or 10% are remotely related

8 or 11% are moderately related

11 or 15% are highly related

The most important issue is the The most important issue is the concern over concern over

the validity of accommodation the validity of accommodation strategies:strategies:

Research findings suggest that providing Research findings suggest that providing accommodations may increase accommodations may increase performance of ELLs/SDs, while also performance of ELLs/SDs, while also benefiting non-ELLs/SDs.benefiting non-ELLs/SDs.There is not enough research support for There is not enough research support for many of the accommodations currently many of the accommodations currently being used in the national and state being used in the national and state assessments.assessments.The only way to make judgments about the The only way to make judgments about the efficiency and validity of accommodations efficiency and validity of accommodations used by states is to use them in used by states is to use them in experimentally controlled studies with both experimentally controlled studies with both ELL/SWD and non-ELL/SWD students. ELL/SWD and non-ELL/SWD students.

Some forms of accommodation strategies, such as the use of a glossary with extra time, raised the performance of both ELL and non-ELL students (Abedi, Hofstetter, Lord, and Baker, 1998, 2000)

ELL students’ performance increased by 13% when they were tested under glossary with extra time accommodation.

While this looks promising, it does not present the entire picture.

Non-ELL students also benefited from this accommodation, with an increase of 16%.

English and bilingual dictionary recipients may be advantaged over those without access to dictionaries. This may jeopardize the validity of assessment.

Concern over the validity of accommodation strategies: Concern over the validity of accommodation strategies: A research exampleA research example

• Linguistic modification of test items is among these accommodations.

• This accommodation also helped students with learning disabilities.

• Thus, an accommodation may have the potential to be effective and valid for both LEP and SD, consequently relevant for LEPWD.

There are, however, some accommodations that help ELL students with their English language needs without compromising the validity of assessment.

Concerns over the validity of accommodation strategies:Concerns over the validity of accommodation strategies:A research findingA research finding

Strategies that are expensive, impractical, or logistically complicated are unlikely to be widely accepted.

Validity: The goal of accommodations is to level the playing field for ELL/SWD, not to alter the construct under measurement.

Consequently, if an accommodation affects the performance of non-ELL/SWD, the validity of the accommodation could be questionable.

Feasibility: For an accommodation strategy to be useful, its implementation must be possible in large-scale assessments.

Concern over the validity of accommodation Concern over the validity of accommodation strategiesstrategies

How validity of How validity of accommodations can be accommodations can be

examined?examined?Only through experimentally controlled research where:

ELL/SWD and non-ELL/SWD students are randomly assigned to experimental and control groups

Both ELL/SWD and non-ELL/SWD students are observed under accommodated and non-accommodated assessments

Can existing data (from national Can existing data (from national and state assessments) be used for and state assessments) be used for examining the effectiveness and examining the effectiveness and validity of accommodations?validity of accommodations?

Effectiveness of accommodations can be Effectiveness of accommodations can be examined only if ELL/SWD students are examined only if ELL/SWD students are randomly assigned to the randomly assigned to the accommodated and non-accommodated accommodated and non-accommodated conditionsconditions

Validity of accommodations can be Validity of accommodations can be examined only if non-ELL/SWD students examined only if non-ELL/SWD students are randomly assigned to the are randomly assigned to the accommodated and non-accommodated accommodated and non-accommodated conditionsconditions

How the validity of How the validity of accommodations can be accommodations can be

examined?examined?

Using existing data?Using existing data?

Through experimentally Through experimentally controlled field study?controlled field study?

Grade 4 Grade 8 Grade 12

Accommodation Number of LEP students

Weighted percentage of assessed LEP

Number of LEP students


Number of LEP students


Total number of assessed LEP students

829 100 651 100 532 1000

Without accommodation

709 85 561 88 474 91

With accommodation 120 15 90 12 58 9 Primary Accommodations

Large print 0 0 0 0 0 0 Extended time 45 5 41 5 32 5 Read aloud 10 2 10 1 1 0 Bilingual Dictionary 5 1 14 2 8 1 Small group 56 7 22 3 15 2 One-on-one 2 0 2 0 0 0 Scribe or computer 0 0 1 0 1 0 Other 2 0 0 0 1 0

LEP Students Assessed With and Without Accommodations, 1998 NAEP Writing Assessment: National Sample, Public and Nonpublic Schools Combined

1 Source: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Writing Assessment

LEP Students Assessed With and Without Accommodations, 1998 NAEP Civic Assessment:

National Sample, Public and Nonpublic Schools Combined

Grade 4 Grade 8 Grade 12

Accommodation

Number of

LEP

students

Weighted

percentage

of assessed

LEP

Number

of LEP

students

Weighted

percentage

of assessed

LEP

Number

of LEP

students

Weighted

percentage

of assessed

LEP

Total number of assessed

LEP students 272 100 216 100 220 100

Without accommodation 240 89 192 90 201 92

With accommodation 32 11 24 10 19 8

Primary

Accommodations

Large print 0 0 0 0 0 0

Extended time 12 4 12 6 8 3

Read aloud 2 1 3 1 0 0

Bilingual dictionary 0 0 1 2 1

Small group 13 5 8 3 7 3

One-on-one 4 2 0 0 2 1

Scribe or computer 0 0 0 0 0 0

Other 1 0 0 0 0

Research findingsResearch findings

• Pitoniak, M., Lutkus, A., Pitoniak, M., Lutkus, A., Cahalan-Laitusis, C., Cook, L. & Cahalan-Laitusis, C., Cook, L. & Abedi, J. Abedi, J. (2005). Are Inclusion Policies and Practices for State Assessment (2005). Are Inclusion Policies and Practices for State Assessment Systems and NAEP State Assessments Aligned?Systems and NAEP State Assessments Aligned?

• Sireci, S. G., Li, S. & Scarpati, S. (2003). Sireci, S. G., Li, S. & Scarpati, S. (2003). The effects of test The effects of test accommodation on test performance: A review of the literatureaccommodation on test performance: A review of the literature (Center (Center for Educational Assessment Research Report No. 485). Amherst: for Educational Assessment Research Report No. 485). Amherst: University of Massachusetts. University of Massachusetts.

• Thompson, S., Blount, A., Thurlow, M. (2002). Thompson, S., Blount, A., Thurlow, M. (2002). A summary of research A summary of research on the effects of test accommodations: 1999 through 2001on the effects of test accommodations: 1999 through 2001 (Technical (Technical Report 34). Minneapolis: University of Minnesota, National Center on Report 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes.Educational Outcomes.

• Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., & Elliott, J. L. (2000). & Elliott, J. L. (2000). Assessment accommodations research: Assessment accommodations research: Considerations for design and analysisConsiderations for design and analysis ( (NCEO NCEO Technical Report 26). Technical Report 26). Minneapolis: University of Minnesota, National Center on Educational Minneapolis: University of Minnesota, National Center on Educational Outcomes.Outcomes.

• Tindal, G., & Fuchs, L. (2000). Tindal, G., & Fuchs, L. (2000). A summary of research on test changes: A summary of research on test changes: An empirical basis for defining accommodations.An empirical basis for defining accommodations. Lexington, KY: Lexington, KY: University of Kentucky, Mid-South Regional Resource Center.University of Kentucky, Mid-South Regional Resource Center.

• Abedi, J., Hofstetter, C., & Lord, C. (2004). Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment Assessment accommodations for English language learners: Implications for accommodations for English language learners: Implications for policy-based empirical research. policy-based empirical research. Review of Educational Research, Review of Educational Research, 7474(1), 1-28(1), 1-28

How the validity of How the validity of accommodations can be accommodations can be

tested in an tested in an experimentally controlled experimentally controlled

condition?condition? LEP/SWD LEP/SWD Status/ Status/ AccommodationAccommodation

AccommodateAccommodatedd

Non-Non-AccommodationAccommodation

LEP/SWDLEP/SWD

Non-LEP/SWDNon-LEP/SWD

Conclusions and Conclusions and RecommendationRecommendation

Accommodations:Must be relevant to the subgroups of students Must be effective in reducing the performance gap between accommodated and non-accommodated studentsMust be valid, that is, accommodations should not alter the construct being measuredThe results could be combined with the assessments under standard conditionsMust be feasible in the national and state assessments

Conclusion

There is not enough research support for many of the accommodations that are currently used in national and state assessments.

The only way to make judgments about the efficiency and validity of these accommodations is to use them in experimentally controlled situations with both ELL/SWD and non-ELL /SWD students and examine their validity and effectiveness under a solid experimental condition.

The results of CRESST studies along with other studies nationwide have provided support for some of the accommodations used for ELL students.

Conclusion cont.Conclusion cont.

Providing a customized dictionary is a Providing a customized dictionary is a viable alternative to providing traditional viable alternative to providing traditional dictionaries. dictionaries. The linguistic modification of test items The linguistic modification of test items that reduce unnecessary linguistic that reduce unnecessary linguistic burdens on students is among the burdens on students is among the accommodations that help ELL students accommodations that help ELL students without affecting the validity of without affecting the validity of assessments.assessments.Computer testing with added extra time Computer testing with added extra time and glossary was shown to be a very and glossary was shown to be a very effective, yet valid accommodation (Abedi, effective, yet valid accommodation (Abedi, Courtney, Leon and Goldberg, 2003) Courtney, Leon and Goldberg, 2003)

Examples of research-supported accommodations:

Conclusion cont.

Without information on important aspects of accommodations such as validity, it would be extremely difficult to make an informed decision on what accommodation to use and how to report the accommodated and non-accommodated results.

It is thus imperative to examine different forms of accommodations before using them in state and/or national assessments.

For more information, For more information, please contact Jamal please contact Jamal

Abedi at:Abedi at:(530) 754-9150(530) 754-9150

or or

[email protected]@ucdavis.edu

Documents

Jamal Abedi University of California, Davis/CRESST Validity, Effectiveness and Feasibility of Accommodations for English Language Learners With Disabilities