M.A.Sc. Thesis Presentation

Automated Reading Assistance Automated Reading Assistance System System

Using Point-of-Gaze EstimationUsing Point-of-Gaze Estimation

Jeffrey J. Kang

Supervisor: Dr. Moshe Eizenman

Department of Electrical and Computer Engineering

Institute of Biomaterials and Biomedical Engineering

January 24, 2006

IntroductionIntroduction

Reading• Visual examination of text

• Convert words to sounds to activate word recognition

We learn appropriate conversions through repetitive exposure to word-to-sound mappings

Insufficient reader skill or irregular spelling can lead to failed conversion: assistance is required

Objective: Develop an automated reading assistance system that automatically vocalizes unknown words in real-time on the reader’s behalf. The system should operate within a natural reading setting.

What We Need To Do — Step 1What We Need To Do — Step 1

1. Identify the word being read, in real-time

2. Detect when the word being read is an unknown word

3. Vocalization of the unknown word

Identifying the Word Being Read Identifying the Word Being Read

Identify the viewed word using point-of-gaze estimation Point-of-gaze is:

• Where we are looking with the highest visual acuity region of the retina

• Intersection of the visual axis of each eye within the 3D scene

• Intersection of the visual axis one eye with a 2D plane

Point-of-Gaze Estimation MethodologiesPoint-of-Gaze Estimation Methodologies

1. Head-mounted 2. Remote (no head-worn components)

Head-mounted Point-of-Gaze EstimationHead-mounted Point-of-Gaze Estimation

Based on principle of tracking the pupil centre, and corneal reflections to measure eye position

Point-of-gaze is estimated with respect to a coordinate system attached to the head

scene camera

eye camera

IR LEDshot

mirror

corneal reflections

pupil centre

Point-of-Gaze in Head Coordinate SystemPoint-of-Gaze in Head Coordinate System

Point-of-gaze is measured in the head coordinate system, and placed on the scene camera image

Locating the Reading ObjectLocating the Reading Object

The position of the reading object is determined by tracking markers

Mapping the Point-of-GazeMapping the Point-of-Gaze

Establish point correspondences from• the estimated positions of the markers in the scene image

• the known positions of the markers on the reading object Homographic mapping of point-of-gaze from scene

camera image to reading object coordinate system

Identify the Reading Identify the Reading ObjectObject

Extract the barcode from the scene camera image to identify the reading object (e.g. page number)

Match barcode to database of reading objects to determine what text is being read

Identifying the Word Being ReadIdentifying the Word Being Read

Using the mapped point-of-gaze, identify the word being read by table lookup

Sample Reading VideoSample Reading Video

Sample Reading VideoSample Reading Video

Mapping AccuracyMapping Accuracy

0 1 2 3 4 5 60

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Mapping Error (mm)

Pro

po

rtio

n o

f Tria

ls

Point-of-Gaze Estimation MethodologiesPoint-of-Gaze Estimation Methodologies

1. Head-mounted 2. Remote (no head-worn components)

O

2D scene objectZ

X

P

C

visual axis

Y

Remote Point-of-Gaze EstimationRemote Point-of-Gaze Estimation

Point-of-gaze is estimated to a fixed coordinate system

• C – centre of corneal curvature

• P – point-of-gaze

IR LEDs

eye camera

computer screen

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Moving Reading CardMoving Reading Card

How can point-of-gaze be estimated to a coordinate system attached to a moving reading object?

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Estimate MotionEstimate Motion

R, T

t0t1

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Use a Scene Camera and TargetsUse a Scene Camera and Targets

t0t1

Scene Camera

H0

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Calculate Two HomographiesCalculate Two Homographies

t0t1

H1

Scene Camera

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Decompose Homography MatricesDecompose Homography Matrices

t0t1

Scene Camera

R0, T0 R1, T1

O

assumed position of 2D scene object

Z

X

P

C

visual axis

P’

true position of 2D scene object

Y

Calculate Motion of 2D Scene ObjectCalculate Motion of 2D Scene Object

t0t1

Scene Camera

R, T

R0, T0 R1, T1

Point-of-Gaze AccuracyPoint-of-Gaze Accuracy

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1

Noise Std. Dev. (pixels)

Err

or

in P

oin

t-o

f-G

aze

(m

m)

What We Need To Do: Step 2What We Need To Do: Step 2

1. Identify the word being read, in real-time

2. Detect when the word being read is an unknown word

3. Vocalization of the unknown word

Dual Route Reading ModelDual Route Reading Model

text

speech

Orthographic Analysis

OrthographicInput Lexicon

PhonologicalOutput Lexicon

Response Buffer

Semantic System

Grapheme-Phoneme

Rule System

LEXICAL ROUTE

NON-LEXICAL ROUTE

Coltheart, M. et al. (2001)

Each word’s graphemes are processed in parallel

Dual Route Reading ModelDual Route Reading Model

text

speech

Orthographic Analysis

OrthographicInput Lexicon

PhonologicalOutput Lexicon

Response Buffer

Semantic System

Grapheme-Phoneme

Rule System

LEXICAL ROUTE

NON-LEXICAL ROUTE

Dual Route Reading ModelDual Route Reading Model

text

speech

Orthographic Analysis

OrthographicInput Lexicon

PhonologicalOutput Lexicon

Response Buffer

Semantic System

Grapheme-Phoneme

Rule System

LEXICAL ROUTE

NON-LEXICAL ROUTE

Each word’s graphemes are individually converted into phonemes based on mapping rules

Detecting Unknown WordsDetecting Unknown Words

For unknown words, the lexical route fails and the slower non-lexical route is used

Hypothesis: we can differentiate between known and unknown words by the duration of the processing time

Processing TimeProcessing Time

Gaze Duration (Subject P.L. - Aloud Reading)

0

500

1000

1500

2000

2500

3000

3500

4000

0 2 4 6 8 10 12

Word Length (letters)

Tim

e (s

ec)

Normal Words

Difficult Words

Setting a Threshold CurveSetting a Threshold Curve

Gaze Duration (Subject P.L. - Aloud Reading)

0

500

1000

1500

2000

2500

3000

3500

4000

0 2 4 6 8 10 12

Word Length (letters)

Tim

e (s

ec)

Normal Words

Difficult Words

Threshold

Threshold curve is a function of word length Model processing time for known words (length k) as a Gaussian random variable

(μk, σk2)

Estimate μk, σk2 from a short training set for each subject

Each point on threshold curve is given by

• α is the constrained probability of false alarm

)μ 2α(1erf2σT k1

kk

Setting the ThresholdSetting the Threshold

Experiment: Detecting Unknown WordsExperiment: Detecting Unknown Words

Remote point-of-gaze estimation system• Reading material presented on computer screen

• Head position stabilized using a chinrest

Four subjects read from 40 passages of text • 20 passages aloud and 20 passages silently

• Divided into training set to “learn” μk, σk2 and set detection

threshold curves

Set false alarm probability α = 0.10 Evaluate detection performance

Experiment: Detecting Unknown WordsExperiment: Detecting Unknown Words

Detection Performance

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4

Subject

Ra

te

Training Set Detection RateTest Set Detection RateTraining Set False Alarm RateTest Set False Alarm Rate

Experiment: Natural Setting Reading Experiment: Natural Setting Reading AssistanceAssistance

Natural reading pose• Unrestricted head movement

• Reading material is hand-held

Head-mounted eye-tracker• Identify viewed word in real-time

• Measure per-word processing time

Detecting unknown words• Processing time threshold curves

established in previous experiment

Assistance• Detection of unknown word activates

vocalization

Experiment: Natural Setting Reading Experiment: Natural Setting Reading AssistanceAssistance

Results

Point-of-gaze mapping method accommodated head and reading material movement without reducing detection performance

Subject Detection Rate False Alarm Rate

M.E. 0.94 0.10

P.L. 0.95 0.09

ConclusionsConclusions

Developed methods to map point-of-gaze estimates to an object coordinate system attached to a moving 2D scene object (e.g. reading card)• Head-mounted system

• Remote system

Developed method to detect when a reader encounters an unknown word

Demonstrated principle of operation for an automated reading assistance system

Future WorkFuture Work

Implement reading assistant using remote-gaze estimation methodology

Validate efficacy of system as a teaching tool for unskilled English readers, in collaboration with an audiologist

Evaluate other forms of assistive intervention

• e.g. translation, definition

Questions?Questions?