Outline
• How does psycholinguistics fit within the umbrella of cognitive science?
• What do we know about language and the brain?
Cognitive Science
Cognitive Computer Cognitive Psychology Linguistics Philosophy Science/AI Neuroscience
Psycholinguistics Natural Language Processing Neurolinguistics
Inter-relationships
What do cognitive psychologists do?
Construct theories that describe how mental processing works in real time:– Within some domain, what are the mental
representations and how are they manipulated?– How does information flow through the mind? – How is processing impacted by memory
constraints, stimulus quality, mode of input, tasks, etc?
Levels of processingSerial vs. parallel processing
Top-down/bottom-upAutomatic vs. strategic processing
What do (formal theoretical) linguists do?
• Construct formal theories of our linguistic knowledge– Sets of rules/principles/assumptions
for generating utterances
• Criteria for a good theory– The rules generate all and only
grammatical outputs (intuitions)– In the simplest way
Derivational Theory of Complexity
• Miller’s clause-processing model of syntactic processing, motivated by Chomsky’s Transformational Grammar.
• Example of taking a linguistic theory and trying to implement it directly as a psychological theory of processing.
Transformational Grammar
• Two levels of syntactic structure, related by transformational rules; accounts for similarity in meaning across different strings.– Surface structure ((John) (picked __ (the box) up)).
– Transformations Particle Movement
– Deep Structure ((John) (picked up (the box))).
• Phrase structure rules and lexical insertion rules used to construct DS
Diff SS, Same DS
Transformations can move, delete, or add words. Complex SS’s require a sequence of transformations– Wh-movement & Subj/Aux Inversion
SS: Which book did Jim buy __?
SS Jim did buy a book.
DS: Jim did buy a/which book
– PassivizationSS: The beer was drunk by Jim.
SS: Jim drank the beer.
DS: Jim drank the beer.
DTC
1.Determine the surface structure of the sentence
2.Reverse the transformations one by one
3.Recover the deep structure
4.Map DS to meaning
Processing Unit is entire Sentence
• Wh-movement & Subj/Aux InversionSS: Which book did Jim buy __?
SS: Jim did buy a book.
DS: Jim did buy a/which book
• PassivizationSS: The beer was drunk by Jim.
SS: Jim drunk the beer.
DS: Jim drunk the beer.
How would you test the DTC?
• Early evidence seemed to support it.
• Later evidence was problematic
• No longer a viable theory of sentence comprehension.
Memory & Language
• Semantic Memory: LTM storehouse of conceptual knowledge
–What is a cow? What is truth?
• Lexicon: LTM storehouse of knowledge about words
–Lexical Entry: For each word/morpheme,Spelling, pronunciation, syntactic category,
pointer to semantic memory
• Episodic Memory: LTM storehouse of our experiences
Spreading Activation Model (Collins & Loftus, 1975)
• Length codes typicality
• Link codes type of relation
• Some redundancy
is a
is
uses
Commmon Current Assumptions about Semantic Memory & the Lexicon
• Semantic Memory is a network of concepts, organized by semantic similarity
• Lexicon is a network of words, organized by phonological similarity
• Interconnections link meanings to words
Cognitive Neuroscience
• Cognitive psychologists tend to talk about the architecture of the mind in terms of functionality– E.g., what is the input to word recognition? What is the output?
• Ultimately, the mental operations described by cognitive psychology occur in the brain
• In some cases, neuroscience can inform cognitive psychology– E.g., we may be able to learn about how words/concepts are
represented by investigating activation patterns in response to different classes of words: action verbs activate motor cortex; perception verbs activate visual cortex
Video Clip (15 min)
Brain Story: First among Equals– The first segment has been comparing human
and chimp abilities to plan. They conclude that chimps can make and execute plans, but not as far into the future as humans.
– We’ll watch a segment on LG• Aphasia patient
• Mapping the brain prior to surgery
What did you learn from the clip?
• What does it mean to have aphasia?
• Do aphasics recover language function? How?
• What is the current view on the role of Broca’s and Wernicke’s areas?
Schiff, N. D. et al. Neurology 2005;64:514-523
Figure 1. Functional maps obtained during listening to narrativesYellow= forward;Blue = backwardRed = both
Averaged healthy data
Schiff, N. D. et al. Neurology 2005;64:514-523
Figure 2. Volumes of activation during the passive listening tasks. The 2 patients are in blue and red; Averaged healthy activation in black.
Schiff et al.
– paper was published shortly after the Terry Schiavo media/political frenzy.
– How do we decide whether a minimally conscious patient is experiencing a life worth living?
• Is language comprehension relevant?
• Why don’t we care as much about how their brain responds to smells?
Outline
• Why is speech perception difficult for computers to do?– Problem of Invariance
• How do humans do it so easily?– Bottom-up information (acoustic signal)– Top-down information (higher level context)
Automatic Speech Recognition Follies (David Pogue, NYTimes, 8/15/02)
What I saidbookmark itMotorollamodem porta procedureand then stick it in the
mailmovie clipsI might addInscrutablethe right or left
What was transcribedbook marketmotor roll amode importupper seizureand dense thicket in the
mailmove eclipseI my dadin screw doublethe writer left
Same sequences of phonemes form different words
Different phonemes, at underlying level
The Problem of Invariance
• Individual phonemes do not have invariant acoustic cues.• There’s a lot of variability in the acoustic signal!• Variance in the acoustic signal has many sources:
– Coarticulation– Differences among speakers– Differences within speakers: yelling/whispering, phone/in-person,
etc.
If the bottom-up acoustic signal doesn’t provide consistent cues, how do we recognize phonemes/words?
If we can’t/don’t rely solely on bottom-up input, how do we
recognize speech?
• Perceive speech as (intended?) articulatory gestures, not as acoustic signal?– McGurk Effect– Sine Wave Speech: There are no essential acoustic
properties that enable speech perception. Rather second-order changes in frequency and amplitude over time are responsible. sine wave speech demos
• Use top-down information (word and sentence context) to complement bottom-up information– If so, when and how?
Is Speech Special?
Modularity Thesis (Fodor, 1983)– The mind is not a unified whole. In addition to
central processes, there are specialized input-output modules
• Central: decision making• Input: color perception, voice recognition• Output: throwing, touch-typing, articulation
– Modules are fast, informationally-encapsulated, mandatory, exhibit characteristic breakdowns, and have shallow outputs.
– Speech perception may be handled by a specialized input module
The Essence of Motor Theory
• Speech perception is grounded in our knowledge of speech production. We recognize phonemes by covertly re-creating the articulatory gestures. (Lieberman et al., 1967)
• Consistent with philosophy that performance & perception are inextricably linked.
• Assume innate, encapsulated phonetic module
Spoken Word Recognition
• Overcoming the problem of invariance in speech perception– Motor Theory– Top-Down Feedback: Word to Phoneme
• TRACE
• Cohort theory of Spoken Word Recognition
Evidence for Top-Down influence on speech perception
• Phoneme Restoration Effect (Warren, 1970)
• Lexical bias in categorical perception task, e.g. dype vs. type (Clifton & Connine, 1987)
TRACE (McClelland & Elman, 1986)
• Interactive connectionist model
• Nodes in network represent phonetic features, segments, & words
• Feature nodes activated by consistent input
• Activation spreads up through network & back down again
• Predicts top-down effects
Is word recognition Automatic & Modular?
Automatic Processes– Fast– Do not require attention– Feed-forward (can’t be guided, controlled,
or stopped midstream)– Not subject to top-down feedback
(informational encapsulation)
Differences between spoken and written word recognition
• For relatively short words, letters in a written word are processed in parallel– Eye movement data– Word superiority effect– Letter-Search Task
• Spoken word unfolds across time– Can recognize some words before they are
completely pronounced.