Upload
john-sparks
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
National Accessible Reading Assessment Projects
Introduction to Accessible Reading Assessment
June 14, 2008
CCSSO National Conference on Student Assessment
Martha L. Thurlow
National Accessible Reading Assessment Projects
Today’s Purpose1. Highlight the challenges in reading
assessment for students with disabilities
2. Provide an overview of relevant research design and analysis
3. Present research findings from projects funded to research and develop accessible reading assessments
4. Identify implications of the research for you
5. Share Principles and Guidelines of based on our research and other resources
2
National Accessible Reading Assessment Projects
National Accessible Reading Assessment Projects
• Designing Accessible Reading Assessments (DARA)
• Partnership for Accessible Reading Assessment (PARA)
• Technology Assisted Reading Assessment (TARA)
3
National Accessible Reading Assessment Projects
NARAP Goals
1. Develop a definition of reading proficiency
2. Research the assessment of reading proficiency
3. Develop research-based principles and guidelines making large-scale reading assessments more accessible for students who have disabilities that affect reading
4. Develop and field trial a prototype reading assessment 4
National Accessible Reading Assessment Projects
Designing Accessible Reading Assessments (DARA)
• Educational Testing Service (ETS)• Focuses on students with learning
disabilities • Focuses on component approach to
assessing reading skills. Primary focus are:– Word Recognition– Reading Fluency– Vocabulary Knowledge– Comprehension
5
National Accessible Reading Assessment Projects
Partnership for Accessible Reading Assessments (PARA)
• Collaboration of National Center on Educational Outcomes and U of MN Department of Curriculum and Instruction, CRESST, U of CA Davis, and Westat
• Focus on all disabilities that impact reading, particularly: – Learning disabilities – Speech or language impairments– Mental retardation– Deafness or hard of hearing
6
National Accessible Reading Assessment Projects
Technology Assisted Reading Assessment (TARA)
• ETS, NCEO and Center for Applied Special Technology (CAST)
• Focus on students with visual impairments• Focus on:
– Examining the performance of operational ELA tests for students with visual impairments
– Development of prototype Technology Assisted Reading Assessment
– Inclusion of VI students in NARAP field test
7
National Accessible Reading Assessment Projects
Background
• All projects focus on standards-based general assessments based on grade-level achievement standards – the regular assessment!
• Not focused on alternate assessments based on alternate achievement standards
• Not focused on alternate assessments based on modified achievement standards
Still, work may sometimes be applicable to these too
8
National Accessible Reading Assessment Projects
Who We Are
Martha Thurlow (PARA, TARA)
Cara Cahalan Laitusis (DARA, TARA)
Linda Cook (DARA, TARA)
David O’Brien (PARA)
Jamal Abedi (PARA)
Discussant – Peggy Carr
9
National Accessible Reading Assessment Projects
Who Are You?
10
National Accessible Reading Assessment Projects
Plan for Today1:00 – 2:30 Introduction
Issues for Students with DisabilitiesResearch Design and Analysis
2:30 – 2:45 BREAK
2:45 – 3:35 Identifying Less Accurately Measured Students
Impact of Motivation and Engagement
3:35 – 3:50 BREAK
3:50 – 4:30 Segmented Reading PassagesPrinciples and Guidelines
4:30 – 5:00 Peggy Carr, Discussant11
National Accessible Reading Assessment Projects
Workshop Notebook
Notebook Tabs – Guide to Workshop Process
• Agenda • Each Topic
• Powerpoint presentation• Resource materials
• Biographies for Presenters• Notepaper
12
National Accessible Reading Assessment Projects
Ground Rules
Ask questions for clarification
Interact with us!
Take care of own needs
13
National Accessible Reading Assessment Projects
Contact Information
Martha L. ThurlowUniversity of Minnesota207 Pattee Hall150 Pillsbury Drive SEMinneapolis, MN 55455
14
National Accessible Reading Assessment Projects
Issues in Assessing Reading of Students with Disabilities
June 14, 2008
CCSSO National Conference on Student Assessment
Martha L. Thurlow
National Accessible Reading Assessment Projects
Poor performance of students with disabilities is a big indicator that there are issues in assessing their reading performance
Data from state reading assessments shows that this is so. . . .
2
National Accessible Reading Assessment Projects
Gaps in Performance on Reading Assessments
Elementary School
3
National Accessible Reading Assessment Projects
Gaps in Performance on Reading Assessments
Middle School
4
National Accessible Reading Assessment Projects
Gaps in Performance on Reading Assessments
High School
5
National Accessible Reading Assessment Projects
Gaps exist in reading performance at all school levels:• They increase as the grade level
increases• They vary by state, but the
variability seems to be more a function of the difficulty of the test than its accessibility (states with the lowest and highest average scores for students with disabilities have smaller gaps – probably due to ceiling and floor effects)
6
National Accessible Reading Assessment Projects
Disabilities affect reading in many ways – we explored the ways in which disabilities may affect reading for 7 categories and developed a report about each:
•Visual Impairments
•Deaf or Hard of Hearing
•Autism
•Learning Disabilities
•Mental Retardation
•Speech or Language Impairments
•Emotional or Behavioral Disabilities
7
National Accessible Reading Assessment Projects
Purpose
To provide general information about specific disabilities and how they interact with reading, so that reading professionals and others who might contribute to the development of accessible reading assessments understand some of the challenges that need to be addressed.
8
National Accessible Reading Assessment Projects
Disclaimer
Papers clearly state that the purpose is to begin a discussion of the issues surrounding reading and students with each disability. The papers were not intended to be comprehensive research reviews.
We have clarified that the papers are for people who do not know the disabilities or for those who have not considered the interaction of disabilities with reading.
9
National Accessible Reading Assessment Projects
Students Receiving Special Education Services
Overview
10
National Accessible Reading Assessment Projects
Reading and Students with Visual
impairments• Most students with visual
impairments are not blind.• Tactile (braille) and auditory
methods of accessing text are most common.
• Common classroom supports and accommodations may not be available for state assessment.
11
National Accessible Reading Assessment Projects
Reading and Students who are Deaf or Hard of Hearing• Age of onset of hearing loss and other
factors shape educational and communication experiences.
• Many communication forms (e.g., American Sign Language, Manually Coded English, lip reading); cochlear implants have raised new issues.
• State assessment policies vary in whether they allow commonly used accommodations. 12
National Accessible Reading Assessment Projects
Reading and Students with Autism
• Many students with Asperger Syndrome can decode words well, but may lack comprehension skills (Barnhill, 2004).
• Students with autism may find it difficult to screen out distractions.
• Accommodations are not as often designated toward this group.
13
National Accessible Reading Assessment Projects
Reading and Students with
Specific Learning Disabilities
• 90% of students with learning disabilities identify reading as their primary difficulty (President’s Commission on Excellence, 2003).
• The read aloud accommodation is one of the most common and controversial accommodations provided for these students.14
National Accessible Reading Assessment Projects
Reading and Students with
Mental Retardation• Historically, educators often skipped academics (including reading) in favor of functional, social, or motor skills.
• Despite wide variety of characteristics that can influence reading (poor short-term memory, low-level meta-cognition), reading skills can be mastered by many students with mental retardation.
• Access to the general curriculum, broader accommodations, and alternate assessments are aspects of reading achievement for students with mental retardation.
15
National Accessible Reading Assessment Projects
Reading and Students with Speech or Language
Impairments• Since reading is a language-based
skill, students without strong language skills may be at-risk.
• Accommodations for these students reflect reading strategies used with them – read aloud, assistive augmentative communication devices, and frequent breaks during assessment.
16
National Accessible Reading Assessment Projects
Reading and Students with Emotional or Behavioral
Disorders• Students need to compensate for lack of attention, distractibility, etc.
• Some accommodations that are needed are generally acceptable (breaks, quite room), while other are questionable (motivational prompts, calming music).
17
National Accessible Reading Assessment Projects
Accommodations
Accommodations are more of an issue for reading assessments than for other content areas.
This occurs because many students use those accommodations that may produce invalid scores (known as modifications in most places).
18
National Accessible Reading Assessment Projects
Percentages of Students Using Certain
Accommodations Not all states publicly report on the use of accommodations, much less the specific accommodations or percentages of students. States that do include:
Colorado North Carolina
Based on 2005-06 data 19
National Accessible Reading Assessment Projects
The Challenge for Accountability
Students who use accommodations that produce invalid scores – modifications – will now count as nonparticipants in the assessment.
20
National Accessible Reading Assessment Projects
Accessibility An accessible assessment is one that reveals the knowledge and skills of students whose characteristics create barriers to accurate measurement of these on traditional reading assessments• It measures the same knowledge
and skills, at the same level• It may reduce the need for
accommodations
21
National Accessible Reading Assessment Projects
Implications• It is important to understand the
characteristics of all students taking assessments, including those that may affect performance but are not what is being measured (e.g., short term memory)
• Disabilities do not are not the “cause” of poor performance – most students with disabilities can perform at levels comparable to their peers – if we make sure they get access to the curriculum, instruction, accommodations, and accessible assessments! 22
National Accessible Reading Assessment Projects
Implications• Accommodations are an important part of
accessibility – more important now than ever before to address them (which can be incorporated into the assessment, which really produce invalid scores)
• We need to explore innovative approaches to improving accessibility – things that in the end may benefit all students
23
National Accessible Reading Assessment Projects
Contact Information
Martha L. ThurlowUniversity of Minnesota207 Pattee Hall150 Pillsbury Drive SEMinneapolis, MN 55455
24
National Accessible Reading Assessment Projects
Research Design and Analysis
June 14, 2008
CCSSO National Conference on Student Assessment
Cara Cahalan Laitusis and Linda Cook
National Accessible Reading Assessment Projects
Overview
• Types of Questions• Research Designs and Analyses• Case Example from DARA project
– Can read aloud and standard scores be reported on the same scale?
• Questions and answers
2
National Accessible Reading Assessment Projects
Research Questions• Are test scores from accommodated and non-
accommodated tests: – Psychometrically comparable?– Measuring the same construct?– Equally valid predictors of the construct?
• What changes to test items (or administration) can: – Increase/decrease accessibility?– Increase/decrease validity of scores?– Engage students with disabilities?– Provide useful feedback to teachers?
3
National Accessible Reading Assessment Projects
Research Studies
• Opinion Research• Item Tryouts• Experimental Studies• Analysis of Operational Test Data
4
National Accessible Reading Assessment Projects
Opinion Research• Types of Opinion Research
– Surveys– Interviews– Focus Groups
• Potential uses: – Explore the types of changes to test items (or
administration) that may be worthy of additional research
– Identify problems in assessment design or administration
5
National Accessible Reading Assessment Projects
Item Tryouts
• Cognitive Labs (Think Alouds)• Pilot Testing• Field Testing
6
National Accessible Reading Assessment Projects
Item Tryouts
• Cognitive Labs (Think Alouds)– 9-20 students per subgroup– Requires one-on-one administration– Qualitative analysis of responses
7
National Accessible Reading Assessment Projects
Item Tryouts
• Pilot Testing– 20-40 students per subgroup– Group administration– Qualitative and Quantitative
analysis of responses
8
National Accessible Reading Assessment Projects
Item Tryouts
• Field Testing– 100 students per subgroup– Group administration– Quantitative analysis of responses
9
National Accessible Reading Assessment Projects
Experimental Studies
• Are test scores from accommodated and non-accommodated tests: – Psychometrically comparable?– Measuring the same construct?– Equally valid predictors of the construct?
• What changes to test items (or administration) can: – Increase/decrease accessibility?– Increase/decrease validity of scores?– Engage students with disabilities?– Provide useful feedback to teachers?
10
National Accessible Reading Assessment Projects
Requirements of Experimental Studies
• Large sample sizes• Random assignment of students to
experimental groups to eliminate– Form effects– Order effects (test form or accommodation)
• Examinations of change generally require:– Two samples (students with and without
disabilities)– Two testing conditions (standard and test change)– Two equated test forms
11
National Accessible Reading Assessment Projects
Pros and Cons Experimental Design
• Pros– Disentangle accommodation or test change from
disability– Impact of test change on total test score can be
directly measured– Impact of other effects (order, test form,
disability-accommodation interactions) can be mitigated
• Cons– Expensive– Time consuming– May not be able to simulate testing environment
for high stakes testing12
National Accessible Reading Assessment Projects
Differential Boost Data Collection Design
Group
Session 1 Session 2
FormAccommodation/
Modification Form
Accommodation/
Modification
1 1 Standard 2Accommodatio
n
2 1 Accommodation 2 Standard
3 2 Standard 1Accommodatio
n
4 2 Accommodation 1 Standard
13
National Accessible Reading Assessment Projects
Analyses• Does test change result in differential
performance gains for students with disabilities?– Repeated Measures Analysis of Variance – Sample sizes vary based on degree of
change expected• Power .8, significance level .05, sample size
needs to be 175 per group to detect and effect size of .20
• Select degree of change that is practically significant rather than just statistically significant:
14
National Accessible Reading Assessment Projects
Analyses
• Which score is a better predictor of construct?– Collect alternative data on construct
• Teachers Ratings • Grades • Alternate test of same construct• Future performance (if test predicts
readiness)
– Analyze data using Regression Analyses• Minimum sample size of 100 per group
15
National Accessible Reading Assessment Projects
Studies Using Operational Data
• Using operational test data to study the validity and fairness of assessments for students with disabilities
16
National Accessible Reading Assessment Projects
Studies Based on Operational Test Data (Overview)
• Using operational test data to study fairness and validity
• Studies that use differential item functioning
• Studies that use factor analysis
17
National Accessible Reading Assessment Projects
Using Operational Test Data to Study Fairness and Validity
• Willingham’s definition of fairness and validity“It seems clear that the overriding issue is
the comparability of tests administered to people with disabilities to those administered to others.”
• Marks of Comparability– Reliability– Factor structure– Item functioning
18
National Accessible Reading Assessment Projects
Using Operational Test Data to Study Fairness and Validity
• Effective program of research might use Willingham’s framework to compare psychometric properties and internal structure of a state test for:– Students without disabilities who take the test
under standard conditions;– Students with disabilities who take the test
under standard conditions; and– Students with disabilities who take the test
with accommodations and/or modifications
19
National Accessible Reading Assessment Projects
Pros and Cons of Using Operational Test Data
• Pros– Readily available– Large sample sizes– Less expensive– Realistic
• Cons– Disability may be poorly or inaccurately
described– Accommodations are bundled; not
always described accurately
20
National Accessible Reading Assessment Projects
Comparing the Internal Structure of an Assessment
• Differential item functioning (DIF)• Factor analysis
21
National Accessible Reading Assessment Projects
What is Differential Item Functioning
• Reference Group– Students whose performance is used as the
standard for the DIF comparison
• Focal Group– Students whose item performance is the focus
of the study
• Test takers matched on proficiency level
22
National Accessible Reading Assessment Projects
Pros and Cons of Using Differential Item
Functioning
Pros• Well established procedures• Can be used with relatively small
samples• Analyses are simple and inexpensive
23
National Accessible Reading Assessment Projects
Pros and Cons of Using Differential Item
Functioning
Cons/issues• Results sometimes difficult to
interpret• Non-uniform DIF• Differences in ability level of focal
and reference groups
24
National Accessible Reading Assessment Projects
Pros and Cons of Using Differential Item
Functioning (cont.)
Cons/issues• Matching criterion• Confounding of disability and
accommodation
25
National Accessible Reading Assessment Projects
What is Factor Analysis
• A statistical method used to explain relationships among variables
• Variables can be test scores or item scores• Exploratory analyses• Confirmatory analyses• Single-group and multiple-group analysis
26
National Accessible Reading Assessment Projects
Pros and Cons of Using Factor Analysis
• Pros– Reduction in number of variables– Identification of groups of
interrelated variables
27
National Accessible Reading Assessment Projects
Pros and Cons of Using Factor Analysis
• Cons– Requires relatively large samples– More than one interpretation can be
made of the same data factor analyzed in the same way
– Factor analysis cannot identify causality
28
National Accessible Reading Assessment Projects
Case ExampleDesigning Accessible Reading
Assessment Project
29
National Accessible Reading Assessment Projects
DARA Project
• Primary Question:– Can we assess some components of
reading (decoding and comprehension) in isolation using multi-stage test design?
• Primary Focus:– Students with learning disabilities,
particularly those who receive read aloud accommodations
30
National Accessible Reading Assessment Projects
Reading Comprehension
Routing Test
Reading FluencyExtended Reading
ComprehensionTest
Decoding and Extended ComprehensionTest with Audio
Extended ComprehensionTest with Audio
31
National Accessible Reading Assessment Projects
Possible Solutions• Increase the reliability of test scores for students
scoring at the chance level (or below) on current state assessments
• Allow students with reading-based disabilities to receive scores on separate components of reading (potentially comprehension, vocabulary, decoding, fluency) as well as total test score for accountability purposes.
• Allows state to count scores of students that receive read aloud modification for AYP because it includes a separate measure of fluency and decoding.
• Holds teachers accountable for both decoding and comprehension instruction.
32
National Accessible Reading Assessment Projects
Research Questions1. Are test scores from standard and read
aloud administrations psychometrically comparable?
2. Does the read aloud administration offer an unfair advantage to test takers with disabilities?
3. Is the “audio + fluency” route comparable to “standard administration” route in terms of predicting teacher’s ratings of reading comprehension?
33
National Accessible Reading Assessment Projects
Research Studies
• Experimentally designed differential boost study– RM ANOVA– Regression– DIF– Factor Analysis
• Analysis of operational test data– DIF– Factor Analysis– Simulation of multistage design
• Cognitive labs
34
National Accessible Reading Assessment Projects
Questions 1
• Does the read aloud administration offer an unfair advantage to test takers with disabilities?
35
National Accessible Reading Assessment Projects
Differential Boost Data Collection Design
Group
Session 1 Session 2
FormAccommodation/
Modification FormAccommodation/
Modification
1 S Standard T Audio
2 S Audio T Standard
3 T Standard S Audio
4 T Audio S Standard
36
National Accessible Reading Assessment Projects
Data Collected
Primary Measure• 2 Reading Comprehension Tests (Form S and
T)– Extra time– Extra time with Read Aloud via CD
Additional Measures• 2 Fluency Measures • 2 Decoding Measures (4th grade only)• Student Survey• Teacher Survey
37
National Accessible Reading Assessment Projects
Sample
• 1181 4th grade students– 527 students with reading based
learning disabilities (RLD)– 654 students without a disability (NLD)
• 855 8th Graders– 394 RLD– 461 NLD
38
National Accessible Reading Assessment Projects
Results of the Differential Boost Study
• RM-ANOVA indicated that students with reading disabilities had a significantly larger boost from an audio (read-aloud) accommodation than students without disabilities – findings consistent with Fletcher et al. 2005 and
Crawford & Tindal, 2004)
• Other Findings– Controlling for other factors (e.g., reading
fluency, decoding)) using RM-ANCOVA does not change these findings
– Controlling for ceiling effects does not change these findings
39
National Accessible Reading Assessment Projects
Questions 2
• Are test scores from standard and read aloud administrations psychometrically comparable?
40
National Accessible Reading Assessment Projects
Analyses of ELA Assessment Using Operational Data
• Analyses– Factor analyses– Differential item functioning analyses and distractor
analyses• Groups of Interest
– Students with learning disabilities who took the test with and without a change in testing conditions
• Test– Grade 4 and grade 8 English-language Arts (ELA)
assessment• Focus
– Determine if the test measures the same constructs for
• Examinees without disabilities• Examinees with learning disabilities who took the test with
and without a change in testing conditions
41
National Accessible Reading Assessment Projects
Number of Items for Grade 4 English-Language Arts Assessment
Test Content No. of Items
Reading Word Analysis, Fluency, and Systematic Vocabulary Development
18
Reading Comprehension 15
Literary Response and Analysis
9
Total—Reading 42
Writing Writing Strategies 15
Writing Applications (Genres and Their Characteristics)
1*
Written and Oral English Language Conventions
18
Total—Writing 34
*Essay item (all others are multiple-choice). The essay item was not used in the study
42
National Accessible Reading Assessment Projects
Differential Item Functioning (DIF) Analyses• The purpose of the DIF study was to examine
whether or not the ELA assessment measured the same construct (s) for the groups in our study
• Used Mantel-Haenszel procedure with total score as criterion
• Mantel-Haenszel categorization– A—negligible DIF– B—slight to moderate DIF– C—moderate to large DIF
• Direction of DIF Flags– Negative favors reference group– Positive favors focal group
43
National Accessible Reading Assessment Projects
Results of the DIF Study
• Fourth grade results– 1 C DIF item, 10 B DIF items– 5 B DIF items were reading items that
favored students with disabilities who took test with read-aloud change in testing conditions
– 5 B DIF items (3 reading and 2 writing) favored students without disabilities
44
National Accessible Reading Assessment Projects
Results of the DIF Study
• Eighth grade results– 1 C DIF item, 7 B DIF items– Five B DIF items (4 reading and 1
writing) favored students who took test with read-aloud change in testing conditions
45
National Accessible Reading Assessment Projects
Factor Analyses of ELA Assessment
Purpose: to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study
• Exploratory analyses (separately in each group)– how many factors
• Confirmatory (multi-group)– Establish base-line model– Confirm number of factors needed to describe
data across all groups
46
National Accessible Reading Assessment Projects
Results of Factor Analysis of ELA Assessment
• Compared the internal structure of the grade 4 and grade 8 ELA assessment– Students without disabilities– Students with disabilities (no test conditions
changes)– Students with disabilities (504/IEP
accommodations)– Students with disabilities (read-aloud change in
testing conditions)
• Results suggest test measures same single dimension for all groups
47
National Accessible Reading Assessment Projects
Questions 3
• Is the “audio + fluency” route comparable to “standard administration” route in terms of predicting teacher’s ratings of reading comprehension?
48
National Accessible Reading Assessment Projects
Predictive Validity of Scores
• Regression analyses were conducted to examine which test scores captured the most variance in teachers ratings of reading comprehension by grade and disability group.
• Tested 4 models:– Standard– Standard + Fluency– Audio– Audio + Fluency
49
National Accessible Reading Assessment Projects
Predictive Validity of Scores
• Tests taken with read-aloud do not predict teachers ratings of reading comprehension as well as tests taken under standard conditions.
• However combining read-aloud scores with reading fluency scores results in equal (or better) predictions of teacher ratings than tests taken under standard conditions.
50
National Accessible Reading Assessment Projects
Lessons Learned and Implications• Reading fluency is an important element of reading
comprehension based on teachers ratings– States should consider administering reading
fluency measure when read aloud is used on reading comprehension assessments
• Scores from standard administration and read aloud condition are fairly comparable psychometrically– States should consider replicating these DIF and
Factor Analyses studies to provide some justification for reporting read aloud and standard scores on the same scale
51
National Accessible Reading Assessment Projects
Lessons Learned and Implications
• 4th and 8th graders have no problems using individualized CD players– States should consider this type of
standardized administration instead of human readers
• Students were better than their teachers in predicting if scores would improve with read aloud – Students should be included in the decision
making process on the use of read aloud accommodations
52
National Accessible Reading Assessment Projects
• Questions?• Contact Information
– Cara Cahalan Laitusis [email protected]
– Linda Cook [email protected]
53
National Accessible Reading Assessment Projects
Identifying Less Accurately Measured
Students
June 14, 2008CCSSO National Conference on
Student Assessment
Ross Moen, Martha Thurlow, Kristi Liu
National Accessible Reading Assessment Projects
• As was previously described, typical reading assessments have limitations for assessing the reading skills of students with disabilities.
• Is accessible reading assessment a way to increase test scores for all students with disabilities?
Higher Scores for All?
2
National Accessible Reading Assessment Projects
Interaction Hypothesis Might Suggest All SWD Scores Rise
Typical Assessment
AccessibleAssessment
Students With Disabilities
Students Lacking Disabilities
3
National Accessible Reading Assessment Projects
• Some students with disabilities already score well despite their disabilities.
• Some students with disabilities truly cannot do what a State’s standards require.– Regardless of where the fault lies -
whether with the instruction, the student or elsewhere – assessments should show if a student cannot do what is required.
But Reality Is More Complicated
4
National Accessible Reading Assessment Projects
Assessment
Scores Should Rise For Some Less Accurately Measured
StudentsClearHigh Scores
ClearLow Scores
QuestionableLow Scores
Less Accurately Measured Students (LAMS)
5
National Accessible Reading Assessment Projects
Sources of Reduced Accuracy
Cheating,Narrow Teaching
to the Test
Lucky Learning,Good Guesses
Bias, Inappropriate
Obstacles
Bad Day,Bad Guesses,Test Taking
Errors
Too HighScores
Too LowScores
SystematicError
RandomError
6
National Accessible Reading Assessment Projects
?
Compare test results with (what?) other information
Match
How Can We Identify Potential LAMS?
MismatchCompare
Match
7
National Accessible Reading Assessment Projects
Compare Tests with Teacher Judgment?
? =
8
National Accessible Reading Assessment Projects
• How well can teachers identify LAMS?– Do they say they can?– Can they distinguish reasons for LAMS?– Can they provide supporting evidence?– Do brief supplemental examinations
match teacher judgments?• What can we learn from teachers’
LAMS?– What do they say they need or want?– What do we observe in assessment
situations?
Teacher Nominations: Study Goals
9
National Accessible Reading Assessment Projects
• Teachers complete LAMS nomination questionnaire – 4th and 8th grade classroom, reading,
English/language arts, special education teachers
• Researchers meet with teachers– Structured interview & examine supporting
evidence
• Researchers meet with students– 4th through 8th grade, native speakers of English– Structured interview and differentiated assessment
Teacher Nominations: Study Procedures
10
National Accessible Reading Assessment Projects
• Two phases separated by adjustments in meeting procedures
• 21 teachers at 10 sites completed LAMS nomination questionnaires on 77 students
• Average “misrepresentation” (1-5): 3.89• First phase, met with 2 teachers and 6
students• Second phase, met with 7 teachers and
17 students – all elementary
Teacher Nominations: Study Results
11
National Accessible Reading Assessment Projects
Reasons for Identifying Students as LAMS
Count* Percentage*
Fluency Limitations Obscure Comprehension Skills
32 41.6%
Some Comprehension Limitations Obscure other Skills
22 28.6%
Test Fails to Reveal Non-Tested Strengths
18 23.4%
Responds Poorly to Testing Circumstances or Materials
31 40.3%
Other 5 6.5%
* Note duplicate counts on 77 students sum to a total count of 108 and total percentage of 140%
12
National Accessible Reading Assessment Projects
Teacher Ratings of Hindrances to Student
PerformanceHardly At All A Little Some
Quite a Bit A Lot Blank Mean
Fluency limitations 3 0 4 6 4 <0> 3.47
17.6% 0.0% 23.5% 35.3% 23.5% 0.0%
Comprehension limitations
0 1 5 7 4 <0> 3.82
0.0% 5.9% 29.4% 41.2% 23.5% 0.0%
Low motivation for the test
7 1 4 1 4 <0> 2.65
41.2% 5.9% 23.5% 5.9% 23.5% 0.0%
Keeping attention focused on the test
3 5 5 2 2 <0> 2.71
17.6% 29.4% 29.4% 11.8% 11.8% 0.0%
Getting worn out by the test
5 4 2 3 3 <0> 2.71
29.4% 23.5% 11.8% 17.6% 17.6% 0.0%
Anxiety 5 3 6 0 2 <1> 2.44
29.4% 17.6% 35.3% 0.0% 11.8% 5.9%
Other: 0 1 0 2 7 <7> 4.50
0.0% 5.9% 0.0% 11.8% 41.2% 41.2%
13
National Accessible Reading Assessment Projects
Student Attitudes Toward
Reading and TestsHardly At All A Little Some Quite a Bit A Lot Blank Mean
How much do your read not for school?
1 4 7 1 3 <1> 3.06
5.9% 23.5% 41.2% 5.9% 17.6% 5.9%
How much do you Like reading?
0 0 9 4 3 <1> 3.63
0.0% 0.0% 52.9% 23.5% 17.6% 5.9%
How hard is reading for you?
3 2 7 4 0 <1> 2.75
17.6% 11.8% 41.2% 23.5% 0.0% 5.9%
How well do tests show your reading?
0 1 6 5 2 <3> 3.57
0.0% 5.9% 35.3% 29.4% 11.8% 17.6%
14
National Accessible Reading Assessment Projects
Student Ratings of What Might Help
Hardly At All A Little Some
Quite a Bit A Lot Blank Mean
Shorter reading passages
0 2 4 7 1 <3> 3.50
0.0% 11.8% 23.5% 41.2% 5.9% 17.6%
More interesting passages
0 3 1 4 6 <3> 3.93
0.0% 17.6% 5.9% 23.5% 35.3% 17.6%
Computer instead of paper and pencil
2 1 2 4 4 <4> 3.54
11.8% 5.9% 11.8% 23.5% 23.5% 23.5%
Entire test read aloud by CD etc
1 1 7 2 3 <3> 3.36
5.9% 5.9% 41.2% 11.8% 17.6% 17.6%
Computer pronounces or explains words you pick
0 0 1 6 7 <3> 4.43
0.0% 0.0% 5.9% 35.3% 41.2% 17.6%
Other ideas you have
0 1 0 1 5 <10> 4.43
0.0% 5.9% 0.0% 5.9% 29.4% 58.8%
15
National Accessible Reading Assessment Projects
Qualitative Analysis:Teachers’ LAMS confirmed?
Clear Bulls EyeConsensus between
researchers & teachern = ?
Off TargetIndications that student is not a
LAMSn = ?
Seems CloseDiffer on why
LAMSn = ?
Seems CloseWeak confirmation
n = ?
16
National Accessible Reading Assessment Projects
For More Information, Contact:
Ross Moen
17
National Accessible Reading Assessment Projects
The Impact of Motivation and Engagement on Assessing Reading
Comprehension
June 14, 2008CCSSO National Conference on Student Assessment
Deborah R. Dillon and David G. O’Brien
National Accessible Reading Assessment Projects
Construct of Comprehension
• Current assessments of reading comprehension are inadequate (Sweet, 2005)
• The current knowledge base on reading comprehension is sizeable but too “sketchy” to provide a foundation for a systematic instructional agenda (RAND Reading Study Group, 2002)
2
National Accessible Reading Assessment Projects
Relation of Motivation and Engagement to Comprehension
• The RAND RSG initiated its work by generating this definition of reading comprehension: “Reading comprehension is the process of
simultaneously extracting and constructing meaning through interaction and involvement with written language” (Snow, 2002, p.11).
• Guthrie and Wigfield (2005) built upon this definition, contending that involvement with written language connotes motivation
3
National Accessible Reading Assessment Projects
Motivation and Comprehension
• Involvement assumes an active, intentional stance toward the text, enabling one to both persevere in getting information from text and using both the textual information and cognitive processes to make meaning
• Without motivation, specifically the intention and persistence to the goal of understanding texts for various purposes, there is little comprehension
• Thus, Guthrie and Wigfield (2005) argued that definitions of reading comprehension should include motivation
4
National Accessible Reading Assessment Projects
Motivation and Comprehension
• Guthrie and Wigfield (1999) outlined a motivational-cognitive model of reading with import for the development of accessible reading assessments.
• In the model they posit that both cognitive and motivational processes influence reading comprehension
• Wang and Guthrie (2004) found that intrinsic motivation for reading was highly predictive of reading comprehension test performance.
• Beliefs about reading and perceptions of oneself as a reader impacts whether students expect reading to be useful and whether they want to be effective readers (Guthrie & Wigfield, 2005; O’Brien & Dillon, 2008).
5
National Accessible Reading Assessment Projects
Motivation Constructs with Potential for Enhancing
AccessibilityGuthrie and Wigfield (2005) argued that the
validity of comprehension assessment can be improved by enabling students to: – read with clearly defined purposes – take positive stances that support self-
efficacy – exert autonomy through choice to better
employ their cognitive competencies in the testing situation.
6
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
Interest and choice to are especially compelling – research indicates that text interest may
be more important for lower achievers than for more proficient readers
– choice may have a greater impact on these readers’ comprehension (deSousa & Oakhill, 1996). However Schiefle’s (1999) findings indicated that the “interest effect” is independent of prior knowledge or verbal ability.
7
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
• This positive impact on comprehension is particularly true if readers perceive the texts to be attractive– appealing visual elements—fonts, illustrations,
layout, – full of interesting details--desirable length and
difficulty level of (Schraw, Bruning, & Svobada, 1995).
• We acknowledge that prior knowledge, which is correlated with both situational interest and reading achievement, may be a confounding variable
8
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
The NAEP Reader Study: The Effects of Choice in Reading Assessment—Results from The NAEP Reader Special Study of the 1994 National Assessment of Education Progress (Campbell & Donahue, 1997)– U.S. Department of Education, Office of
Educational Research and Improvement ((NCES 97-491) special study
– Examined the feasibility and measurement impact of offering grade 8 and grade 12 test takers a choice of reading material on an assessment of reading comprehension.
9
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
– In the design, a group of readers who could exert choice in selecting from among seven stories to read as part of the 1994 NAEP, was compared with a group who were assigned stories.
– In the choice condition, the researchers found no significant effect for choice for twelfth graders and slightly lower performance in the choice condition for eighth graders
10
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
Our Expansion on the NAEP Reader Study– Assessment matching the goals of the 2009
NAEP Framework in terms of the text types and the cognitive targets assessed (Reader study used assessed more generic reading comprehension constructs following all passages)
– Using both literary-fiction and informational-exposition passages (Reader study used narrative only)
– Items following the reading of passages match the content and key ideas of individual passages
11
National Accessible Reading Assessment Projects
Interest and Choice Enhance Accessibility
• Respondents exert choice “choose your own assessment” before assessment is administered (Reader study included reading passage summary and choice as part of testing time)
12
National Accessible Reading Assessment Projects
Calibration Study
The purpose of the study is to scale or calibrate the measurement tools that will be used in a large-scale accessible reading assessment for students with disabilities.
This process allows investigators to empirically determine the comparability of passages and items used in the reading assessment study by placing all passages and questions on a common IRT (item response theory)-based equal-interval measurement scale.
13
National Accessible Reading Assessment Projects
Research Questions1. What is the difficulty of each reading passage
(based on a passage total score, which, in turn, is based on performance on all passage comprehension items/questions) and each comprehension item/question?
2. How well can the reading passages be placed on a common interval measurement scale to allow scores from different passages (of equal or unequal difficulty) to be compared and equated?
3. Based on IRT item fit statistics, what multiple choice items should be retained and which should be eliminated?
4. Which reading passages do students prefer to read?
14
National Accessible Reading Assessment Projects
Design
• Representative total sample of 1200 students, representing a range of reading ability and including students with disabilities for 4th and 8th grade
• Selection of 40 Passages, including 10 literary/fiction and 10 informational/exposition administered five forms
• 10 items for each passage using 2009 NAEP cognitive targets
15
National Accessible Reading Assessment Projects
Analysis
• This preliminary item/passage psychometric calibration study allows for:
1.the placement of all passages/questions on a common equal-interval measurement scale,
2.the development of passage scoring tables by which to assign subjects reading “ability” scores, and
3.provision of a mechanism for equating scores across different passages.
4.This “item fit analysis” will determine which items will be retained and those that will be eliminated.
16
National Accessible Reading Assessment Projects
Motivation “Choice” Study
Purpose: To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement.
17
National Accessible Reading Assessment Projects
Research Questions
1. Is there an interaction effect between choice, type of text, and type of student?
2. Is there a correlation between general motivation to read and performance on a large-scale comprehension assessment?
3. Are participants who are more motivated to read more likely to benefit from the choice option on a large-scale assessment?
4. Does the option of exercising choice improve comprehension for general education students and for students with disabilities?
18
National Accessible Reading Assessment Projects
Participants and Design
• Students fluent in English (140 4th graders; 140 8th graders)
• Targeted samples representing range of disability groups
• Random assignment to treatment (motivation-choice) and control (no choice) reading 2 literary fiction and 2 informational-expository passages followed by 5-6 multiple choice items
• Untimed administration• Assessment of general and situational
motivation 19
National Accessible Reading Assessment Projects
Design & Procedures cont.
• Post-assessment interviews conducted with subsets of students from the control and experimental groups at both grade levels.
• Students from the various disabilities groups as well as regular education students selected for interviews (16 students from 4th grade and 16 from 8th grade)
20
National Accessible Reading Assessment Projects
Analysis
A split-plot design will be used – two between-subjects factors (A =
passage choice & B = disability status), – one within-subjects factor (C = text
type), one blocking variable (S = subject), & one covariate (X = motivation as assessed on the MRQ) at the between-subject level (A, B, C, and X are fixed effects, and S is a random effect)
21
National Accessible Reading Assessment Projects
Analysis--contd.
• Analysis of variance will be used to evaluate various effects; correlations of students’ performance on the comprehension test & responses on the MRQ and situated motivation questions will be calculated
• Various analytic deduction approaches will also be used to analyze the post assessment interview data and a mixed-design approach will be used to integrate the overall quantitative and qualitative findings
22
National Accessible Reading Assessment Projects
Implications• Motivation can lead to increased engagement,
which can lead to higher, more valid comprehension performance on high stakes assessments
• In constructing “bias-free” assessments, test designers may eliminate engaging passages or design features either unwittingly or in the interest of psychometrics
• Students with disabilities may be more likely to engage with, persevere with, and exert more effort with more motivating and engaging passages, defining these assessments as more accessible than typical reading assessments
23
National Accessible Reading Assessment Projects
Contact Information
Deborah R. Dillon University of Minnesota, Twin Cities330B Peik Hall, 159 Pillsbury Dr. S. E.Curriculum & Instruction DepartmentUniversity of Minnesota, Twin CitiesMinneapolis, MN 55455 email: [email protected]
David G. O’Brien125 Peik Hall, 159 Pillsbury DriveCurriculum and Instruction DepartmentUniversity of Minnesota, Twin CitiesMinneapolis, MN 55455email: [email protected]•
24
National Accessible Reading Assessment Projects
Segmented Text Study
June 14, 2008
CCSSO National Conference on Student
Assessment
Jamal Abedi, Seth Leon & Jenny Kao
National Accessible Reading Assessment Projects
Purpose
•To determine if reducing the length of reading passages by segmenting them would impact performance of student’s with disabilities.•To examine the impact of segmenting on students’ non-cognitive domains such as anxiety, fatigue, frustration and motivation.
2
National Accessible Reading Assessment Projects
Background of Study
• From “chunking” to “segmented text”– Chunking in past literature deals with working
memory capacity, with the hypothesis that reading material chunked into meaningful units facilitates reading comprehension and efficiency.
– However, chunking in the literature refers to chunking sentences.
– We use “segmented text” to refer to how passage segments are grouped with their corresponding items on the test page.
3
National Accessible Reading Assessment Projects
Segmented Text
• Segmented text also serves as “built-in” test breaks, possibly reducing the need for accommodation
4
National Accessible Reading Assessment Projects
Participants
• 738 Grade 8 students from ten public schools in California
• 620 non-SD, 117 SD • Of the 117 SD:
– 107 specific learning disabilities– 2 deaf/hard of hearing– 3 autistic– 2 speech/language impairment– 4 other health impairments
5
National Accessible Reading Assessment Projects
Reading Test
• Three reading comprehension passages were obtained from publicly-released tests from two states.
• Two versions of the test were created: Original (version A) and Segmented (version B)
• Test designed to be completed in one classroom period (approx. 50 min.)
6
National Accessible Reading Assessment Projects
Passages
• All passages were informational (i.e., not fiction or literature).
• First passage was 700 words, other two passages were about 550 words each.
• Each passage had 8 multiple-choice items with 4 possible answer choices (24 total MC items).
7
National Accessible Reading Assessment Projects
Process of Segmenting
• Segments were grouped with corresponding test items
• Each passage was broken down into 3 to 4 segments; each segment contained 1-3 questions
• Inferential questions appeared at the end
• Test items appeared in the same order in both versions
8
National Accessible Reading Assessment Projects
Other Instruments
• Teacher Ratings• Emotion/Mood Inventory• Motivation Scale
9
National Accessible Reading Assessment Projects
Teacher Ratings
Asked teachers to rate each of their students. Corresponds with Calif. (CST) proficiency levels.
In your opinion, how would you rate this student’s reading comprehension ability?• Advanced• Proficient• Basic• Below basic• Far below basic
10
National Accessible Reading Assessment Projects
Emotion/Mood Inventory
Asked students after each passage:How does taking the test make you feel? Please circle all the
words that describe how you feel. There is no right or wrong answer.
If none of these words describe how you feel, please circle NONE.
good tired
energetic upset
bored confident
frustrated okay
happy stressed
blanked out interested
relaxed badNONE
11
National Accessible Reading Assessment Projects
Motivation Scale
• Post-test (printed at the end of the test booklets)
• 10-item, 4-point Likert-type, combining “importance” and “effort” questions
12
National Accessible Reading Assessment Projects
Research Questions
• Accessibility– Segmented Text and Reliability– Segmented Text and Performance– Segmented Text and the correlation between
teacher ratings, English language arts (ELA) achievement test level and reading performance
• Affective Factors– Segmented Text and Motivation– Segmented Text and Emotion/Mood Inventory
13
National Accessible Reading Assessment Projects
Segmented Text and Reliability Findings
• The original version of the assessment was more reliable for non-SDs than for SDs.
• This reliability gap decreased on the segmented version (no longer a significant difference).
• This suggests the segmented version may be more accessible for SD students
Reliability limits validity, because rxy < √ rxx’ (Allen & Yen, p. 113)
Groups Reliability Validity
SD/Original
(n=53)0.516 .718
SD/Segment
(n=62)0.689 .830
Non-SD/Original
(n=312)
0.783 .884
Non-SD/Segment
(n=305)
0.788 .888
14
National Accessible Reading Assessment Projects
Segmented Text and Performance
• No significant differences in reading performance of either group due to segmenting
Groups Mean SD n
SD/Original 9.94 3.32 52
SD/ Segment
9.32 4.05 57
Non-SD/ Original
13.89 4.58 301
Non-SD/ Segment
13.88 4.67 292
15
National Accessible Reading Assessment Projects
Motivation Results
Summary of descriptive analyses for the motivation section
No significant differences
Group Mean SD n
Students with disabilities, original 22.21 3.65 53
Students with disabilities, seg. 22.83 3.44 60
Students with disabilities, total 22.54 3.54 113
Non-disabled, original 21.36 5.07 313
Non-disabled, seg 22.16 4.23 296
Non-disabled, total 21.75 4.69 609
Original version, total 21.48 4.89 366
Segmented version, total 22.27 4.12 356
Total 21.87 4.54 722
16
National Accessible Reading Assessment Projects
Conclusions
• Results suggest segmented text may be more accessible to students with disabilities
• Segmenting did not affect performance of non-SD students; therefore, it did not alter the reading construct
• Segmenting did not affect performance of SD students either
• Segmenting improves psychometric characteristics of reading comprehension assessments
17
National Accessible Reading Assessment Projects
Implications for the large scale reading assessments
• The study could help states in identifying factors that affect accessibility of reading assessment
• It provides methodological paradigm for the study of accessibility of reading
• It examined important factors that affects presentation of test items
• Since segmenting passages improves the reliability of reading assessment without altering the construct, states can apply this feature into their assessments.
• This study encourages states and test publishers to look into test characteristics in a more comprehensive way to identify factors affecting accessibility of assessments for all students particularly for students with disabilities.
18
National Accessible Reading Assessment Projects
Accessibility Principles for Reading Assessments
June 14, 2008
CCSSO National Conference on Student Assessment
Martha L. Thurlow
National Accessible Reading Assessment Projects
Purpose of the Accessibility Principles
To identify supported* principles and guidelines for making large scale assessments of reading proficiency more accessible for students who have disabilities that affect reading, while maintaining a high level of validity for all students taking the assessments.
Support = research, standards, and theory
2
National Accessible Reading Assessment Projects
“Accessibility”
An accessible assessment is one that reveals the knowledge and skills of students whose characteristics create barriers to accurate measurement of these on traditional reading assessments
3
National Accessible Reading Assessment Projects
Intended Use
• Large-scale reading assessments• Assessments focused on grade-level
content standards based on grade-level achievement standards (regular or alternate)
• Reading portion of normative assessments included in state assessment programs
• Reading component of English language proficiency assessments4
National Accessible Reading Assessment Projects
Review Process
• NARAP Technical Advisory Committees• NARAP Principles Committee• Workshop at annual conference of the
Association of Test Publishers• Interactive session at National
Conference on Student Assessment*
* Attend interactive session on Monday, June 16 at 8:15 am (Florida Ballroom I)
5
National Accessible Reading Assessment Projects
Organization of Accessibility Principles
• Principles are “rules” that define the overarching goals to achieve accessibilityA rationale is provided for each
principle to justify why it is included• Guidelines under each principle
address the implementationSupport is provided for each
guideline via an annotated bibliography 6
National Accessible Reading Assessment Projects
Overview of Principles
1. Reading assessments should be accessible to all students in the testing population, including students with disabilities
2. Reading assessments should be grounded in the field of reading
3. Reading assessments should be developed with accessibility as a goal throughout rigorous and well-documented test design, development, and implementation procedures7
National Accessible Reading Assessment Projects
Overview of Principles
4. Reading assessments should reduce the need for accommodations, yet be amenable to accommodations that are needed to make valid inferences about a student’s performance
5. Reporting of reading assessment results should be designed to be transparent to relevant audiences and to encourage valid interpretation and use
8
National Accessible Reading Assessment Projects
Contact Information
Martha L. ThurlowUniversity of Minnesota207 Pattee Hall150 Pillsbury Drive SEMinneapolis, MN 55455
9