Applying New Voice Recognition Technology to Formative

AssessmentMargaret Heritage

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

Markus IseliHenry Samueli School of Engineering, UCLA

CRESST Conference,Los Angeles, CA

September 8th, 2005,

Overview

• Project Aims and Components

• Features of the Program

• Automatic Speech Recognition Technology

• Interface Demo

• Assessment Framework

• Looking to the Future

Project Components• Develop speech recognition technology for

children

• Apply technology to create an on-demand, easy-to-use system of assessment in reading for students in grades K-2

• Develop system capacity to present auditory, text, graphical stimuli, and to score, analyze and adapt to responses

• Develop query-based data mining to monitor students’ progress

• Develop easy-to-understand displays of data analysis

Specific Aims

Develop assessment system that :

Is helpful for teachers (i.e. has instructional utility and saves time)

Reduces variability (e.g., consistent instructions, consistent delays, consistent scoring)

Automatically scores and analyzes children’s performance on reading assessment tasks

Distinguishing Features

• Strong interdisciplinary interactions among electrical engineering, computer science, education, psychology and linguistics

• Collaboration with expert teachers

• Focus on bilingual (Mexican-Spanish accented English) students

• Validation of the system

Lens of Project

• Information that teachers can use next day in their instruction?

• Sensitivity to English Language Learners(ELLs)

• Sensitivity to Language Knowledge

Instructional Utility

Effective classroom is assessment-centered (NRC, 2000)

• ongoing assessment of students’ learning that provides the

day-to-day fuel for instruction

Formative assessment

• ‘ used to adapt the teaching work to meet the learning needs’

(Black, Harrison, Lee, Marshall, & Wiliam, 2003, p.2).

Reading error or pronunciation difference?

In Spanish, compared to English

Phonetics p t k closer to Eng b d g than to p t k

s z n t d: tongue on teeth, not behind them

Sounds missing: th, oy, etc.

Phonology s+ptkbdg only across syllables

Distinctions like ‘bit-beat’ not made

Literacy Words spelled ‘y’ pronounced ‘j’, (by some)

Words spelled ‘i’ pronounced ‘ee’, etc.

Academic Language

Among the factors contributing to non-comprehension of text:

• inadequate knowledge of the words used,

• lack of familiarity with the syntactic structures

(Lyon, 1998)

Automatic Speech Recognition (ASR)

• Teach the computer to understand human speech

OR• Teach the human being how to talk

to be understood by a computer

Three main challenges:• Speaker: gender, pronunciation,

health, dialect, language

• Environment: noise, other speech

• System: devices, program

Child vs. Adult Speech

• Children don’t talk just to be understood by the computer, they just talk!

• Children cannot yet control their articulators as well as adults

• Children have different anatomical features (shorter vocal tract), and these features change fast

• Children have very high pitch frequency

Designing an ASR System

Questions

• Who is going to use the ASR system?

• In what environment will it be used?

Implementation

• Collect “a lot” of appropriate data

• Train the system

• Test the system and make changes

Our Assessment SystemThe system will:

• Assess each student in the same manner (no dependence on teacher)

• Produce visual stimuli (letters, words, phrases) and record the child’s vocal response

• Measure response times very accurately

• Analyze the recorded audio and other data to generate reports which are useful to teachers (ASR)

• Be easily accessible (internet)

• Handle multiple users at the same time

System Architecture

Interface

Client Side Server Side

More detailed…

Flash Interface: Live Demo

http://kittychan.icsl.ucla.edu/tball

Flash interface design by Larry Casey

System Design BenefitsAccessible

Software: common web browser Hardware: standard microphone

Flexible Command structure is open-ended Allows for any audio-visual testing set-up

Stable Constant audio stream: everything is

captured Stimulus/response data is recorded in real

time

Scalable Content, display, navigation are

independent

Assessment Framework

• Recall the lens

• Guiding questions:

Are the assessments embedded in an instructional framework?

What is the instructional value of the information?

How much assessment is too much?

Assessment Framework

• Guiding questions privilege a hierarchical rather than a uniform approach to assessment.

• All students take benchmark assessments as a check on progress

• Some students take 'drill down' assessments related to specific skills on an as-needed based for diagnosis

• Teachers have guidance on what to assess

Assessment Framework

Skills Assessed:

• Phonemic Awareness

• Word recognition

• Oral reading (accuracy and rate)

• Comprehension

• Syntax

Assessment Framework

Links to:

• English Language Development Standards

• English Language Arts Standards

Narrative Oral

Reading

NarrativeReading

Comp

Narrative Oral

Reading

NarrativeReading

Comp

Basic Monitoring Spine of Reading Assessment

Framework

Narrative Oral

Reading

NarrativeReading

Comp

The Dam (decodable word list)

K/1 High Frequency Word List

Rapid Naming

LetterSound

Comp.

BPST

Narrative Oral

Reading

NarrativeReading

Comp

Narrative Listening Comp.

After student demonstrates

mastery of listening

comprehension and word reading

tasks, begin assessing in

connected text skills.

After student demonstrates

mastery of letter sound and naming tasks, begin assessing

regular and irregular word

reading.

1. 2. 3. 4.

Basic Reading Assessment Framework - Kindergarten

Narrative Oral

Reading

NarrativeReading

Comp

Begin the framework with

screening assessments in

listening comprehension, letter sound,

and naming tasks .

Repeat assessing

connected text skills

throughout year as needed.

If problem arises, recheck word reading development

I.I.

I.I.

Reading Assessment Framework with Interventions - Kindergarten

Phonemic Awareness

I.I. I.I.

I.I.

The Dam (decodable word list)

I.I. I.I.

Irregular Word list

Narrative Listening Comp.

The Dam (decodable word list)

K/1 High Frequency Word List

Rapid Naming

LetterSound Comp.

BPST

If student demonstrates skill

level below mastery, provide instructional

intervention and reassess.

Narrative Oral

Reading

NarrativeReading

Comp

Continue assessing

connected text skills

throughout year as needed.

Vocab and Topic

Knowledge

I.I. I.I.

Written Lang.Comp.

(Syntax)

Narrative Oral

Reading

NarrativeReading

Comp

Oral Lang. Comp.

I.I.

I.I.Vocab and

Topic Knowledge

Oral Language

Comp.

San Diego High

Frequency Word List

BPST

Narrative Oral

Reading

NarrativeReading

Comp

Repeat assessing connected text

skills throughout year as needed. If

problem with progress arises, recheck word

reading development.

After student demonstrates mastery

of listening comprehension, letter

sound, and word reading tasks, begin

assessing in connected text skills.

Begin the framework with screening assessments in

listening comprehension,

letter sound, and word reading tasks .

1. 2. 3.

1st and 2nd Grades

Narrative Oral

Reading

NarrativeReading

Comp

San Diego High

Frequency Word List

BPST

Narrative Listening Comp.

Reading Assessment Framework with Interventions- First and Second Grades

The Dam (decodable word list)

Irregular Word list

I.I. I.I.

Vocab and Topic

Knowledge

Written Lang.Comp.

(Syntax)

I.I. I.I.

Vocab and Topic

KnowledgeOral Lang.

Comp.(Syntax)

I.I. I.I.

I.I.

The Dam (decodable word list)

K/1 High Frequency Word List

I.I.

Phonemic Awareness

I.I.

Rapid Naming

I.I.

LetterSound Comp.

I.I.Continue assessing connected text skills throughout year as

needed.

Narrative Oral

Reading

NarrativeReading

Comp

ExampleKindergarten/1st grade BPST

Blending Words with:

Short map rip met rub mop

Final-e fine rope rake cute kite

Long soap leak pain feed ray

r-control fur sort sir tar serve

OVD coin moon round lawn foot

2 syllable silent ladder napkin locate cactus

Reporting

Looking to the Future

• Validation of assessment system

• Development of ASR to include discourse level performances

• Leverage other CRESST technology( QSP)

• Applications to other domains ( e.g., math and science)

• Applications to other grade levels