Technology Assisted Reading Assessment National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of TARA project

Technology Assisted Reading Assessment

National Accessible Reading Assessment ProjectsGeneral Advisory Committee

December 7, 2007

Overview of TARA projectPsychometric workTest Development

Elizabeth Stone Survey and Interviews of Teachers of Students with

Visual Impairments (TVIs) Martha Thurlow


Overview of TARA project



• Focus on improving state reading assessments for students with visual impairments

• Jointly directed by Cara Cahalan-Laitusis (ETS) and Martha Thurlow (NCEO)

• Subcontract to Center for Applied Special Technology (CAST)


Project Tasks

• Examining the performance of operational ELA tests for students with visual impairments

• Development of prototype Technology Assisted Reading Assessment

• Contribute research findings to NARAP Principles and Guidelines

• NARAP/TARA Field Test with VI students


Psychometric work


Nature and Purpose of DIF and DDF Studies

• Evaluate comparability of measurement characteristics for students without disabilities and students who are blind or visually impaired– No reason to consider overall abilities of

groups to be different– Look for construct-irrelevant causes

• A priori DIF hypotheses (position, visual content, textual content, language, format)


Description of Test

• ELA component of large-scale state standards test

• Focused analyses on 4th and 8th grades

• 75 multiple choice questions per grade– 56% Reading / 44% Writing– Essay component in 4th grade, excluded

from analyses


Description of Samples:Summary Statistics

Large PrintLarge Print or

Braille

Reference Focal Focal

4 N 30225 105 141Mean 48 37 37SD 14 15 15

8 N 30069 74 98Mean 46 34 35SD 12 12 14

GradeSummary Statistics

Students who are Blind or Visually Impaired (all;

English Language Learner + non-ELL)

Students without

Disabilities (sample; non-

ELL only)


Methods of Analysis• DIF (Differential Item Functioning): See if groups

seem to get item right or wrong in different proportions after being matched on ability– Mantel-Haenszel with purification– ELA score as matching criterion– ETS delta-DIF categories (-C, -B, A, B, C)

• DDF (Differential Distractor Functioning): See if groups seem to choose distractors in different proportions after being matched on ability– Standardization method– Typical cut-offs for significance


DIF Results Summary

• A priori hypotheses were not supported• Number of DIF items smaller than expected

(sample size issue?)• Reading/Writing divide by grade

– Most grade 4 Reading DIF items favored the focal group (4/5)

– Most grade 4 Writing DIF items favored the reference group (4/5)

– Most grade 8 Reading DIF items favored the reference group (5/8)

– All grade 8 Writing DIF items favored the focal group (2/2)


DIF Results Summary• Where DIF did seem to show up:

– Metaphor– Items and passages related to how people

experience the world (e.g. through touch)– Unusual document format

• Where DIF did not seem to show up:– Items and passages involving typically

“sighted” activities or interests, e.g. photography (in fact, some favor focal groups)

– Effects did not seem to apply for all items associated with a passage


Favors Students without Disabilities

Favors Students who are Visually Impaired using Large Print

Easy

Difficult 0.00

0.25

0.50

0.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cen

t co

rrec

t Reading A

Writing A

Reading B

Writing B

Writing C

Grade 4Reference: Students without disabilities

taking standard formFocal: Students who are visually impaired taking large

print form



Favors Students who are Blind or Visually Impaired using Large Print or Braille

Easy

Difficult 0.00

0.25

0.50

0.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cen

t co

rrec

tReading A

Writing A

Reading B

Writing B

Grade 4Reference: Students without disabilities taking

standard formFocal: Students who are blind or visually impaired taking large print or braille form


Grade 8 Reference: Students without disabilities

taking standard formFocal: Students who are visually impaired

taking large print form


Favors Students who are Visually Impaired using Large Print

Easy

Difficult0.00

0.25

0.50

0.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cen

t co

rrec

tReading A

Writing A

Reading B

Writing B


Grade 8Reference: Students without disabilities

taking standard form Focal: Students who are blind or visually impaired taking large print or braille form


Favors Students who are Blind or Visually Impaired using Large Print or Braille

Easy

Difficult 0.00

0.25

0.50

0.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cen

t co

rrec

t Reading A

Writing A

Reading B

Writing B

Reading C


DDF Results (grade 4 only)

Results were not as interpretable as hoped– For the large print focal group (10 B or C DIF

items): 2 items had a highly significant distractor; 8 items had at least one moderately significant distractor

– For the large print or braille focal group (5 B or C DIF items): 1 item had a highly significant distractor; 3 items had at least one moderately significant distractor

– Review did not reveal any obvious causes


Review of Forms

Large print form issues: – Portions were merely an enlargement, so

some fonts, e.g. footnotes, were not in proper font size.

– As with the standard form, many different fonts were used. This is not considered “best practice” for large print.

– We noted one passage description that was slightly different, and one item that had different wording from the standard form.


Review of Forms

Braille form issues: – Distracting logo encoding and page numbering– Paragraph numbering– Contracted vs. uncontracted– Symbols used

• Symbols not encountered before (e.g., italics)

• Nemeth code vs. literary braille


DiscussionThere are some caveats:

– Sample sizes smaller than desired: These are “low incidence” populations.

– We would need more of each type of item to support inference of a pattern.

However, some inferences can be drawn that may be useful in instruction, test development, administration, and research:– access to curriculum– access to various document formats for test preparation– careful review of test forms for these issues– importance of proctor awareness– need to include these groups in fairness and validity

measures


Presentations and Publications

The DIF and DDF work has been presented at:– Association of Test Publishers (ATP) February 2007 – Association for Supervision and Curriculum

Development (ASCD) March 2007– National Council on Measurement in Education (NCME)

April 2007 – Council for Exceptional Children (CEC) April 2007– Institute of Education Sciences (IES) Research

Conference June 2007– Northeastern Educational Research Association (NERA)

October 2007

The TARA DIF work is to be published as an ETS Research Report (RR) and will then be able to be found using:

http://search.ets.org/custres/


Future Steps for TARA psychometric work

• Examine items and item types across administrations

• Investigate other DIF methods• Test analysis• Trend analysis• Contribute test development

recommendations• Creation of TARA using ECD


Test Development


Evidence-Centered Design Structure

– Purpose of Test– High Level Claim– Population of Test Takers– Test Structure– Proficiency Levels


Elements of TARA via ECD

– Purpose of Test• accountability assessment for instruction of

technology assisted reading• evaluate a student’s readiness to participate in the

regular state assessment for English language arts using assistive technology accommodations

• does not replace state reading assessment

– High Level Claim• measure the degree to which a student can

independently access grade level English language arts text using assistive technology


– Population of Test Takers• Students who are blind or visually impaired (using

IDEA 2004 definition) are a heterogeneous group who access text or printed materials in a variety of formats often using a wide assortment of assistive technology. Of these students, the test will have the following audience:

– Students in grades 7 to 10 whose primary method of reading includes assistive technology

– Students whose IEP includes instruction in reading with assistive technology

– Not appropriate for students with significant cognitive impairments who participate in the state’s alternate assessment based on alternate achievement standards

– Not appropriate for students who do not have at least literal comprehension skills as measured by a 5 question screening test



Elements of TARA via ECD– Test Structure

• Section 1: Literal Comprehension (Screening)• Section 2: Using Assistive Technology (Access)

– Screen reader tasks would be » adjusting the speed at which material is read, » moving by word or sentence, » accessing embedded links, etc.

– Screen-magnification tasks might include » changing the magnification level of the screen, » adjusting the colors, » finding particular portions of the material on-screen.

– Other tasks will involve » locating material that is specified literally, e.g. “the second

paragraph of section three” or “the sentence reading, ‘Morse developed the telegraph.’,”

» opening and closing documents » locating structural elements (index, glossary, a particular word in

the glossary or a particular cited work in a bibliography).


– Proficiency Levels

• Advanced: Student can access grade-level text independently and efficiently, with satisfactory literal comprehension, using one or more forms of assistive technology.

• Proficient: Student can access grade-level text independently, with satisfactory literal comprehension, using one or more forms of assistive technology.

• Below Proficient: Student has, at most, limited ability to access text independently using one or more format of assistive technology or has less than satisfactory literal comprehension when using assistive technology



Test Development Timeline• Draft Evidence Centered Design (ECD) Guide

– Population of Tests Takers– Score Reports– Proficiency Models

• Draft ECD Models and Test Blueprint based on research findings (Spring 2008)– Evidence Models,– Task Models and Task Shells, – Assembly Models, and– Scoring Models

• Develop pilot test items (Summer 2008)• Pilot test (Fall 2008)• Revise and assemble prototype assessment (Spring 2009)• Field test prototype assessment (2009-10)


Survey and Interviews of Teachers of Students with Visual Impairments (TVIs)


Survey

Conducted: May 11-25, 2007

Purpose: To obtain information from teachers of students who are blind or have visual impairments – information about current platforms, reading approaches, and other aspects of reading for these students.


Target Sample – TVIs

• Typically certified special education teachers with extensive coursework and professional development experiences in communication skills, braille instruction, access to assistive technology, and providing support to general education classroom teachers

• Typically work in state schools for the blind or in one or more regular public schools; usually work across grades, and have wide variety of duties and responsibilities


Methods• 25 question survey

• Piloted by peers in assessment and visual impairment

• Revisions made to ensure logic and ease of use for both sighted and respondents with visual impairments

• Survey provided online and via paper

• Volunteer survey respondents selected for interviews


Coverage of Survey Items

• Demographic information on TVI• Information about students on

caseload• Information on instruction• Information on assessments


Responses• 185 online responses• 12 paper responses

Total responses = 197 responses

(Participation request sent to listservs of AER Div 17, AFB lists, & NFB)

30% return rate if no overlap and all received request - unlikely)


Analyzed Responses• Focused on TVIs with caseloads that

included students in grades 7-10

• Final count = 146

• Those not included may not have had any students in grades 7-10 or may not have been TVIs.

• Number of responses per item varied, down to as low as 98


Summary of TARA Survey Results

• TVIs had an average of 12.4 years of experience (range = 0.33 - 40 years; median = 8.5 years)

• Average caseload of students in grades 7-10 was 5.8 (overall caseload average = 16.1 students)

• Respondents spent an average of 35% (median 30%, mode 50%) of their instructional time using computer software assistive technology

• Primary goals most often cited for instructional time were “become a proficient user of assistive technology (42%), and “read using a combination of approaches” (30%)


Summary of TARA Survey Results• Respondents spent an average of:

– 27% of reading instruction time on direct instruction of how to use assistive technologies to assist in reading

– 19% of time in supported reading aloud– only 9% of time in direct instruction of phonemic strategies

(Braille or print)

• Survey data showed that most students had congenital vision loss (81%) rather than adventitious (19%)

• Most students (80%) also have an additional disability documented on their IEP– largest among them cognitive impairment (28%), physical

impairment (17%) and learning disability (16%)



• The largest percentage of students (28%) receive their services in a general classroom with itinerant support, or a general class room with resource room support (23%), and few receive services at a school for the blind (10%)

• Students access print through visual (25%) or visual + audio (29%) a majority of the time

• A majority of students (96%) use some kind of accommodation or assistive technology at times in the classroom, largest among them:– audio (38%), large print (35%), read aloud (26%), and

braille (25%)



• Students with visual impairments use JAWS for Windows (26%), ZoomText Magnifier (13%), Duxbury (13%), and ZoomText Magnifier/Reader (10%) to access text most often

• A positive correlation occurred between the number of years spent as a TVI and the % of students using tactile + audio to access print (p=.010)

• The % of students whose primary goal is to become a proficient user of assistive technology correlates positively with TVIs spending a student’s instructional time using computer software assistive technology (p<.001)


Summary of TARA Survey Results• An inverse relationship between the size of the

caseload and the percent of students using these accommodations: – Braille (p=.044)– Audio (p=.004)– CCTVs (p=.030)– Screen Readers (p=.003)

• The % of students whose primary goal is to read using a combination of approaches is inversely correlated to the relative importance to the students TVI of sounding out words (p=.011)


Implications of Survey Results

• AT use is a large part of instruction for students with VI

• TVIs take a “blended” approach to teaching reading (e.g., using a variety of modalities)

• Standardization of assessments may be a challenge because students use different technology for different purposes


TARA Interview Status

• Interview sample consisted of TWIs from a variety of settings (Schools for the Blind, resource rooms, itinerant)

• By end of November, 2007, 27 interviews had been completed.


Preliminary Findings• Other assessments from which to build

TARA assessment (e.g., Texas School for the Blind)

• Assistive technology use appears to be dependent on motivation of teachers and students

• In many states, the only option for large-scale assessment is large print or Braille


Preliminary Findings• Non-portability of some technologies make

it difficult for students to practice at home or in some settings (e.g., community settings, different classrooms)

• Product selection often depends on teacher knowledge of product (teachers are more willing to recommend products with which they have familiarity)

Documents

Technology Assisted Reading Assessment National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of TARA project