Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension...
35
Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language Technology www.pprice.com
Technology Based Assessment of Language and Literacy Patti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language
Technology Based Assessment of Language and Literacy Patti
Price - Intro - Decoding - Comprehension - Opportunities/Discussion
PPRICE Speech & Language Technology www.pprice.com
Slide 3
Primates Gorillas Humans Stone tools FireShelter Spears
Language: ~ 50,000 years ago Clothing Burial Hunting Art Farming,
10,000 years ago Bronze, 5500 years ago Written language: ~5 6,000
years ago Intro Decoding Comprehension Discussion
Slide 4
Pre-language: Information access by direct experience Oral
language: Information access also from someone in same time and
place with same language Written language: Information access not
limited by time and place Multimedia: The power of direct
experience, but not limited by time, place, language, learning
style, sensory abilities Intro Decoding Comprehension
Discussion
Slide 5
Written and Spoken Language are Different These differences are
important in learning to read. How difficult it is to spell (many
letter combos to one sound) How difficult it is to read (many
sounds for the same letter) How closely your dialect matches
written conventions Got to say this for you, you got guts. Guts and
no brains. But guts alone dont mean nothin. I gotta say this for
you-- you have guts. Guts and no brains. But guts alone mean
nothing. Ive got to say this for you: You have guts. You have guts
and no brains, but guts alone mean nothing. Intro Decoding
Comprehension Discussion
Slide 6
Differences Between Speech and Writing Vocabulary Distinct
words Sentence length Sentence structure Grammar Prosody
WrittenSpoken things stuff 20,000+ 2,000+ 10 30 words ? Complex
shallow Conservative Liberal A., ! ? ; : () Rich Intro Decoding
Comprehension Discussion
Slide 7
~130 words (1) Ive been talking Ive been multiplying matrices
already, but certainly time for me to discuss the rules for matrix
multiplication. (2) And the interesting part is the many ways you
can do it, and they all give the same answer. (3) So its and theyre
all important. (4) So matrix multiplication, and then, uh, come
inverses. (5) So were uh, we mentioned the inverse of a matrix, but
theres thats a big deal. (6) Lots to do about inverses and how to
find them. (7) Okay, so Ill begin with how to multiply two
matrices. (8) First way, okay, so suppose I have a matrix A
multiplying a matrix B and giving me a result well, I could call it
C. (9) A times B. Okay. (10) Uh, so, l- let me just review the rule
for w- for this entry. Transcript from a linear algebra lecture
(From Glass, Hazen, Lee and Wang, Analysis and Processing of
Lecture Audio Data, 2004) Example ~45 words Weve been multiplying
matrices. Now lets discuss the rules for matrix multiplication.
Note that there are many ways to do it and get the same answer. One
way to multiply two matrices, A and B, to get a new matrix C is the
following rule: Intro Decoding Comprehension Discussion
Slide 8
Overview of TBALL T echnology B ased A ssessment of L anguage
& L iteracy Child-friendly testing system Measure & analyze
child reading & pronunciation 5-8 year olds Native speakers of
English & Spanish Multidisciplinary collaboration: o
Engineering o Education o Computer science o Linguistics o
Neuroscience UCLA, USC, UCB Intro Decoding Comprehension
Discussion
Slide 9
8 TBall Goals and Components Reliable and objective automatic
reading assessments Comprehensive instructional framework for
Diagnosis Intervention Three main components: Childrens interface
Teacher interface Assessment module o Decoding (DAM) o
Comprehension (BARLA) Intro Decoding Comprehension Discussion
Slide 10
9 Decoding Comprehension Need to be balanced In alphabet-based
languages, hints about pronunciation If you know the language you
are reading, the hints may be enough If your first language is
another language, and/or if you know the LTS rules in another
language, there may be problems. MOSTLY depends on successful
decoding Much more to learn than LTS rules Intro Decoding
Comprehension Discussion
Slide 11
Decoding Necessary (but not sufficient) for reading
comprehension Necessary for reliable and efficient processing of
text Strongly related to speed of initial reading acquisition A
predictor of difficulties in comprehension development Accounts for
much variance in reading ability at all ages Intro Decoding
Comprehension Discussion
Slide 12
English Letter to Sound Rules Example: pronunciations of o
Pronunciations of the letter o observed in the 100 most frequent
words of English 9 OW: go, home, most, no, only, over, so (know,
own) 6 UW: into, to, who, do, (too, you) 6 AH: from, of, other,
another, (some, come) 6 AO: for, or, on, often, (your, more) 5 OU:
(about, hour; down, how, now) 3 UH: (good, goodbye would) 3 -:
(people, work, world) 2 AA: got, not 1 W AH: one Intro Decoding
Comprehension Discussion
Slide 13
Written Language, Complexity LanguageLetterSound Englishc/s/,
/k/ hot, car, walk, ah/aa/ Spanisha/aa/ Frencha/aa/ allo, pot,
eau/ow/ SpellingSound ache + past = ached/ey k/ + /t/ aim + past =
aimed/ey m/ + /d/ paint + past = painted/p ey n t/ + /ax d/ Intro
Decoding Comprehension Discussion
Slide 14
Decoding Assessment Are the words read correctly? Decodability:
closer to one letter one sound is better In sample of 100 most
frequent words in grade 1 material m, j, b, d pretty straight
forward (always /m jh b d/) Digraphs (2 letters -> 1 sound):
-ng, -ck, (wh-) 2 letters -> 2 sounds (or): th -> /dh, th/
(thy, thigh) 1 letter -> 2 sounds (and): x -> /k s/ (fox,
box) G: get, thing, laugh, large E: because, get, been, they,
great, often O: go, do, from, for, how, good, people, got, one
Intro Decoding Comprehension Discussion
Slide 15
Decoding Assessment Some words are more decodable than others
High frequency words tend to be less decodable High frequency words
tend to be sight words (not decoded but recognized instantly) (this
is important) Decoding skills are also important If a word is
well-known by a child, it may be treated as a sight word and NOT
decoded (tyrannosaurus) Therefore, unless we know which words are
sight words for a child, its hard to assess decoding skills
Nonsense words must be decoded Decoding Assessment Measure Intro
Decoding Comprehension Discussion
Slide 16
Decoding Issues: Dialect, Idiolect and Assessment If the child
has trouble with the letter 'r' (common in English), and says 'cow'
when shown 'car' is it a misreading? If the child is from Boston
and reads 'Carl' as 'Call' is it a misreading? What if the teacher
has never heard anyone from Boston? if the child sees the word
'ask' and says 'aks'? if the child is a native speaker of another
language, e.g., Spanish, and says 'seat' when presented the word
'sit', is it a misreading? if the child is already learning to
read, e.g., Spanish, and sees the word 'sore' and pronounces it
'sore-ay', is it a misreading? SOLUTION: Map the child's system,
whatever it is Augment the dictionary to diagnostic labels. Intro
Decoding Comprehension Discussion
Slide 17
Dialect Modeling in ASR Dialect is a system The system is not
well modeled in ASR These are being merged These are being merged
When the point is the Contrast system When the point is the
Contrast system Head sections adapted from Gunnar Fants Acoustic
Theory of Speech Production Intro Decoding Comprehension
Discussion
Slide 18
Slide 19
Decoding Issues: Dialect, Idiolect and Assessment Grape g r ey
p Grape g w ey p Grape g r aa p ey Bird b er d Bird b uh d Bird b
iy r d Rocks w aa k s Horse h ow s Three th w iy Run w ah n Intro
Decoding Comprehension Discussion
Slide 20
Decoding Results nug: /n uw g/ - Sp. LTS, phonology, or
short/long? jore, tay, bap: /hh ao r ey, t ay, b aa p/ - Sp. LTS
yan, vag, hine: /zh aa n, b aa g, ay n/ - Sp. Phon. rin, bap, mot:
/r iy n, b ey p, m ow t/ - short/long jore: /zh ow r ey/ - mixed
Spanish phonology, LTS jick: /hh ih s k/ - Spanish LTS, misapplied
English LTS Intro Decoding Comprehension Discussion
Slide 21
Comprehension Assessment GRADE 2: Kim has a dog. The dog's name
is "Dot." Dot is a very white dog, and he has a black dot on his
leg. Sam, Ben, and Kim like to play with Dot. They play ball with
Dot all day long. Ben lost the ball in the mud, and Dot went into
the mud to get the ball. Now, Dot is a brown dog from the mud! What
does Dot look like? A Dalmatian Brown I dont remember Like a dog A
white dog A white dog with a black dot on his leg White and a dot
on his leg Well hes white and he has a brown dot on his leg Like a
rat Please describe Dots physical appearance? Intro Decoding
Comprehension Discussion
Slide 22
Motivation/ Desiderata Equivalent measures for reading or
listening (to compare) Separate understanding of probe from
understanding of text (probe as picture adds interest and is not
text or speech) Diagnose reading error patterns (by selection of
alternate items) Easy to automate (compared to most comprehension
tests) Items ascend in difficulty (several errors in a row stops
the test) Many items (to allow retesting without repeating items)
Equal number of plausible/implausible items (to mitigate guessing)
PIAT-R (Peabody Individual Achievement Test Revised) is best match,
but Not automated Reading only They did not respond to our request
Intro Decoding Comprehension Discussion
Slide 23
Intro Decoding Comprehension Discussion Vocabulary Syntax
Morphology Accent adjustment Prosody Find words Print adjustment
Punctuation Decoding (LTS) In your first language, this might be
easier In a new language, this might be easier
Slide 24
BARLA Example 1. (pot and box reversed) 2.(dog in box) 3.(box
on boy) 4.(correct) The pot is on a box. Hear OR Read: The pot is
on a box. Intro Decoding Comprehension Discussion
Slide 25
BARLA Example Here is a fire. 1.tire (rhymes) 2.fire (correct)
3.fur (vowel differs) 4.fairy(shares, f, i, r) Hear or read: Here
is a fire. Intro Decoding Comprehension Discussion
Slide 26
Decoding Opportunity We sampled the space with 25 items High
Frequency Word: tell, nap Low Frequency Word: fell, zap Nonsense
Word: zell, fap However, desirable would be Frequent retesting
(perhaps in a game environment) Diagnostic of which decoding skills
need more work Items need to be generated and coded for diagnosis
Intro Decoding Comprehension Discussion
Slide 27
An Opportunity! Intro Decoding Comprehension Discussion
ContentGamePedagogy User Model Letter sounds, English: a /ae/ b /b/
c /k/, /s/ z /z/ By font, audio files Levels: Lower case simple to
complex Upper case Digraphs Move up if: 100% correct Move down if:
3 wrong in a row Last played Sounds mastered Diagnosis Preferences
Engine b p d b p d
Slide 28
An Opportunity! Intro Decoding Comprehension Discussion
ContentGamePedagogy User Model Vocabulary, English: bat Levels:
Lower case simple to complex Upper case Digraphs Move up if: 100%
correct Move down if: 3 wrong in a row Last played Sounds mastered
Diagnosis Preferences Engine b p d b t a
Slide 29
A Comprehension Opportunity! Data collected for sentences
analogous to PIAT-R But not analyzed Paradigm ready to hatch more
uses For more information about the project:
http://nautilus.icsl.ucla.edu/tball/ For more information about the
data: [email protected][email protected] Intro Decoding Comprehension
Discussion
Slide 30
An Opportunity! Intro Decoding Comprehension Discussion Natural
language processing and reading augmentation
Slide 31
Simplify Vocabulary By "augmenting human intellect" we mean
increasing the capability of a man to approach a complex problem
situation, to gain comprehension to suit his particular needs, and
to derive solutions to problems. Increased capability in this
respect is taken to mean a mixture of the following: more-rapid
comprehension, better comprehension, the possibility of gaining a
useful degree of comprehension in a situation that previously was
too complex, speedier solutions, better solutions, and the
possibility of finding solutions to problems that before seemed
insoluble. And by "complex situations" we include the professional
problems of diplomats, executives, social scientists, life
scientists, physical scientists, attorneys, designers--whether the
problem situation exists for twenty minutes or twenty years. We do
not speak of isolated clever tricks that help in particular
situations. 1. Select Text2. Select View3. View By "augmenting
human intellect" we mean increasing the capability of a man to
approach a complex problem situation, to gain comprehension to suit
his particular needs, and to derive solutions to problems.
Increased capability in this respect is taken to mean a mixture of
the following: more-rapid comprehension, better comprehension, the
possibility of gaining a useful degree of comprehension in a
situation that previously was too complex, speedier solutions,
better solutions, and the possibility of finding solutions to
problems that before seemed insoluble. And by "complex situations"
we include the professional problems of diplomats, executives,
social scientists, life scientists, physical scientists, attorneys,
designers--whether the problem situation exists for twenty minutes
or twenty years. We do not speak of isolated clever tricks that
help in particular situations. Simplify vocabulary Simplify syntax
Simplify both Simplify vocabulary Simplify syntax Simplify both By
"adding to people's brains" we mean adding to the way we deal with
difficult things, to add to understanding according to our needs,
and to figure out solutions to problems. Intro Decoding
Comprehension Discussion
Slide 32
Simplify Syntax By "augmenting human intellect" we mean
increasing the capability of a man to approach a complex problem
situation, to gain comprehension to suit his particular needs, and
to derive solutions to problems. Increased capability in this
respect is taken to mean a mixture of the following: more-rapid
comprehension, better comprehension, the possibility of gaining a
useful degree of comprehension in a situation that previously was
too complex, speedier solutions, better solutions, and the
possibility of finding solutions to problems that before seemed
insoluble. And by "complex situations" we include the professional
problems of diplomats, executives, social scientists, life
scientists, physical scientists, attorneys, designers--whether the
problem situation exists for twenty minutes or twenty years. We do
not speak of isolated clever tricks that help in particular
situations. 1. Select Text2. Select View3. View By "augmenting
human intellect" we mean increasing the capability of a man to
approach a complex problem situation, to gain comprehension to suit
his particular needs, and to derive solutions to problems.
Increased capability in this respect is taken to mean a mixture of
the following: more-rapid comprehension, better comprehension, the
possibility of gaining a useful degree of comprehension in a
situation that previously was too complex, speedier solutions,
better solutions, and the possibility of finding solutions to
problems that before seemed insoluble. And by "complex situations"
we include the professional problems of diplomats, executives,
social scientists, life scientists, physical scientists, attorneys,
designers--whether the problem situation exists for twenty minutes
or twenty years. We do not speak of isolated clever tricks that
help in particular situations. Simplify vocabulary Simplify syntax
Simplify both Simplify vocabulary Simplify syntax Simplify both By
"augmenting human intellect" we mean increasing the capability of a
person to approach a complex problem situation. We also mean how a
person gains comprehension to suit his particular needs and derives
solutions to problems. Intro Decoding Comprehension Discussion
Slide 33
Simplify Vocab & Syntax By "augmenting human intellect" we
mean increasing the capability of a man to approach a complex
problem situation, to gain comprehension to suit his particular
needs, and to derive solutions to problems. Increased capability in
this respect is taken to mean a mixture of the following:
more-rapid comprehension, better comprehension, the possibility of
gaining a useful degree of comprehension in a situation that
previously was too complex, speedier solutions, better solutions,
and the possibility of finding solutions to problems that before
seemed insoluble. And by "complex situations" we include the
professional problems of diplomats, executives, social scientists,
life scientists, physical scientists, attorneys, designers--whether
the problem situation exists for twenty minutes or twenty years. We
do not speak of isolated clever tricks that help in particular
situations. 1. Select Text2. Select View3. View By "augmenting
human intellect" we mean increasing the capability of a man to
approach a complex problem situation, to gain comprehension to suit
his particular needs, and to derive solutions to problems.
Increased capability in this respect is taken to mean a mixture of
the following: more-rapid comprehension, better comprehension, the
possibility of gaining a useful degree of comprehension in a
situation that previously was too complex, speedier solutions,
better solutions, and the possibility of finding solutions to
problems that before seemed insoluble. And by "complex situations"
we include the professional problems of diplomats, executives,
social scientists, life scientists, physical scientists, attorneys,
designers--whether the problem situation exists for twenty minutes
or twenty years. We do not speak of isolated clever tricks that
help in particular situations. Simplify vocabulary Simplify syntax
Simplify both Simplify vocabulary Simplify syntax Simplify both By
"adding to people's brains" we mean adding to how we deal with
difficult things. We also mean adding to understanding that is
suited to our needs, and to how we can figure out solutions to
problems. Intro Decoding Comprehension Discussion
Slide 34
Simplify Vocabulary & Syntax: Result By "adding to people's
brains" we mean adding to how we deal with difficult things. We
also mean adding to understanding that is suited to our needs, and
to how we can figure out solutions to problems. By adding to what
we can do, we mean a mix of several things. Faster and better
understanding are examples. We also mean making it possible to get
some understanding when before it was too difficult. And, we mean
faster solutions, better solutions, and being able to find
solutions that didn't seem possible before. By "difficult
situations" we mean the problems people have in different kinds of
jobs. The jobs can be in the government, running businesses, being
scientists, lawyers, or designers. The problems can be there for
twenty minutes or twenty years. We do not mean clever tricks that
help in only some situations. We mean a way of life. In this new
way of life many things are useful. Useful things include hunches,
cut-and-try, things we can't touch, and our "feel for a things.
These things are more useful with great ideas, ways to talk about
the ideas and ways to write them, intelligent ways of doing things,
and powerful technologies to help us. The number of people and what
they create are growing fairly quickly. But our problems are
getting harder even more quickly. Also, the importance of finding
solutions becomes greater because things are changing more quickly
than they used to change. Things that before happened only in one
place now are spreading to the whole world. This also makes it
important to find solutions. If we could make our brains better in
this way, any intelligent person would run as fast as possible to
get it. To make them want to run, we need to show a way to do it.
We also need to show what we might get out of it. Intro Decoding
Comprehension Discussion
Slide 35
Summary of Views and Reading Ease Original Words239 Sentences7
Word/sentence34.1 Characters/ word 5.6 Reading Ease10.7%
Flesch-Kincaid Grade Level 12.0 Vocabu -lary SyntaxBoth 275249315
71522 39.216.614.3 4.75.64.4 42.3%28.2%71.7% 12.0 6.8 Intro
Decoding Comprehension Discussion