Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
WP4 Language Emergence
Britta Wrede (BIEL)Katharina Rohlfing, Karola Pitsch,
Katrin Lohan, Lars Schillingmann, Sascha Griffiths, Gerhard Sagerer, BIEL
Jun Tani, RIKENStefano Nolfi, CNR
Angelo Cangelosi, Martin Peniak PLYMChrystopher Nehaniv, Kerstin Dautenhahn, Yo Sato, Joe Saunders,
Frank Förster, Caroline Lyon, UHKerstin Fischer, Arne Zeschel, USD
Overview
Task 4.1• Generalization as a basis for emergence
of symbolic systems (start: M7)Task 4.2
• Acoustic Packaging and the learning of words (start: M13)
Task 4.3• From single word lexicons to
compositional languages (start: M13)Task 4.4
• Constructional grounding and primary scenes (start: M19)
Task 4.5• Evolutionary origins of action and
language compositionality (start: M31)
ITALK Year 3 Review Genoa, 21. June 2011
Progress in WP4 in Y3• 4.1 Language learning with MTRNN:
– Generalisation wrt noise, syntactic category, sentence complexity
• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-grained
synchrony– Results on sync between syntax and motion
• 4.3 Word order facilitates– Learning syntactic categories and related actions– And to generalise
• 4.5 Origins of compositionality– Action compositionality to learn language compositionality in
interaction
• 4.4 Construction learning– Grammar induction– Higher level grounding in infants: Semantic salience (internal
reasoning) and input frequency (tutor input)
ITALK Year 3 Review Genoa, 21 June 2011
ActionActionHierarchy 4.1
AcousticPackages 4.2
Objectives & Goals
Speech
GrammaticalConstructions
4.3
4.4
Lexicon Construction
ITALK Year 3 Review Genoa, 21. June 2011
Evolutionary Originsof Compositionality 4.5
ActionActionHierarchy 4.1
AcousticPackages 4.2
Objectives & Goals
Speech
GrammaticalConstructions
4.3
4.4
Lexicon Construction
ITALK Year 3 Review Genoa, 21. June 2011
4.1 Generalization as a basis forthe emergence symbolic system
RIKEN
ITALK Year 3 Review Genoa, 21. June 2011
Objective
• Learning of complex sentences by MTRNN
• Analysis on effects of noise in generation and learning.
• Analysis on internal trajectories of representing sentences
W. Hinoshita, H. Arie, J. Tani, H.G. Okuno, T. Ogata: "Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network", Neural Networks, Vol.24, pp.311-320, 2011
ITALK Year 3 Review Genoa, 21. June 2011
MTRNN model
IO vector ( 30 dimensions) 6 dimensional parametric vector P( = Cs Initial states)
IO : 30τ = 2
□a b z ,. ?c ・・
・
・
Cf : 40τ = 5
Cs: 11τ = 70
( = white space )
Example for training
• Verb
• Noun : ball, box• Article : a, the• Adverb : slowly, quickly• Adjective
intransitive : run, walk, jumptransitive : touch, kick, punch
size : big, smallcolor : yellow, blue, red
7 categories, 17 words
Experimental procedure1. 100 sentences are generated from the CFG.2. 80 out of 100 are trained for MTRNN.3. Evaluate MTRNN recognition capability by utilizing
the 100 sentences.
a. Recog:Sentence Cs Init Vectorb. Gene:Cs Init Vector sentencec. Compare:Target sentence in (a) and the generated
one in (b)
Eval
If correctly generated, adequate structure is self-organized.
-60 -50 -40 -30 -20 -10 0 10-15
-10
-5
0
5
10
15
Cs: Initial State Analysis
PCA1
PCA
2
Simplicity
Simplicity in objective phrase
No adjectiveEx: Punch a box.
Two adjectivesEx: Kick a big red ball. One adjective
Ex: Touch a red ball.
adverb
Yes
NoIntransitiveEx: Walk slowly.
Summary
• Functional hierarchy: Alphabet, word, sentence
• Generalization capability: – Recognize unlearned sentences
• Auto-correction of incorrect sentences– Positive effects by learning through noisy inputs
• Sentence structures appear in init state space of slow context units.– Complexity of objective phrases (w/wo adjectives)– w/wo adverbs– Intransitive versus transitive sentences
ITALK Year 3 Review Genoa, 21. June 2011
4.2 Acoustic Packaging
Bielefeld University
ITALK Year 3 Review Genoa, 21. June 2011
Acoustic Package
Acoustic Package
t
t
Speech
Motion
A Computational Model ofAcoustic Packaging
Goals
• Acoustic packaging makes use of interaction between modalities at an early processing level
• Evaluation showed AP is able to reflect differences between adult-adult and adult-child interaction
• Goals Y3 + Y4– Using Acoustic Packaging for word learning– Providing learning units for further processes– Generating feedback events
ITALK Year 3 Review Genoa, 21. June 2011
Cues Focusing on Details in the Interaction
Cues Especially Relevant for Temporal Action Segmentation
Additional Cues and their Role in Acoustic Packaging
Acoustic Packaging
Speech Activity
Motion Segmentation
Finding emphasized syllables:Acoustic Prominence
Trajectory extraction:Motion/Color Saliency
Detecting Moving Colored Objects
•Detecting changing regions
•Clustering in YUV color space
•Ranking according to color distance (U,V) to centroid of all clusters
•Heuristical filtering•Trajectory accumulation
ITALK Year 3 Review Genoa, 21. June 2011
Acoustic Prominence
• Goal: Relative ranking of syllables emphasis within an utterance
• Syllable Segmentation– Mermelstein algorithm
• Features for Emphasis Rating[Tamburini, Wagner 2007]
– Nucleus duration– Spectral emphasis– Pitch movements– Overall intensity
• Currently used: Spectral emphasis• Example with 3 syllables context
Und zum Schluss packen wir noch den roten Becher in den gelben Becher
Finally we put the red cup in the yellow cup
Prominence
roten
Feedback based on Acoustic Packages
• iCub replays prominent syllables
• iCub replays trajectories• Acoustic packages simplify
access to corresponding multimodal events at a time
• Example 1 (implemented)– iCub replays syllables
associated with a specific cup color triggered by showing the cup to iCub
• Example 2 (in progress)– iCub replays trajectories
associated with a specific (prominent) syllable triggered by speech
Acoustic Package
Speech Interval
Motion Segments
Trajectories
Syllable Segmentation
Prominence Ranking
ITALK Year 3 Review Genoa, 21. June 2011
Prominent Syllable –Trajectory Color Association
(en)
ITALK Year 3 Review Genoa, 21. June 2011
iCub‘s Perspective
Synchrony betweenverbal utterances and action
• Semantic: stressed color label (“red”, “yellow”, “green”) and color of detected trajectory
• Temporal: distance of prominent syllable to nearest max trajectory change
ITALK Year 3 Review Genoa, 21. June 2011
Synchrony betweenverbal utterances and action (USD, BIEL)
• Results: Sig. Correlations between– Syntactic category and prominence level– Syntactic category and movement velocity– Pause length and movement velocity
• Further Questions– Synchrony inside vs outside AP– Syntactic structure inside vs outside AP
ITALK Year 3 Review Genoa, 21. June 2011
4.3 From single word lexicons to compositional languages
University of Plymouth
ITALK Year 3 Review Genoa, 21. June 2011
Word Order
• As a structural cue for– category information, e.g. the N, look at the N– semantic roles, e.g. John kisses Mary
• children are sensitive to such cues (e.g.Gomez 2007)
• children use this information for learning (e.g. StClair et al. 2010)
ITALK Year 3 Review Genoa, 21. June 2011
Word Order Learning
Adjective – Noun Constructionstouch green ball touch green cubetouch red balltouch red cube
Hypothesis: Word order providesinformation about
grammatical category (adjective - noun)semantic category (colour - shape)
ITALK Year 3 Review Genoa, 21. June 2011
Recurrent Neural Network
Arm joint / Touch / Shape / Colour
5 Inputs -language
Arm joint / Touch / Shape / Colour
5 Inputs -language
10 Hidden units
5 Inputs – rec. Lang.
ITALK Year 3 Review Genoa, 21. June 2011
• The robot is trained with only objects and colours in isolation
Experiment I: Synonyms
By analysing internal representations no distinction between name and colours are observed
t1: “TOUCH” input t3: Action
TOUCH BALLTOUCH GREENTOUCH CUBETOUCH RED
Input for Word Order Learning
• Words are input to the neural network in sequence
“TOUCH BALL” (t1) TOUCH(t2) BALL
“TOUCH GREEN” (t1) TOUCH(t2) GREEN
“TOUCH RED CUBE” (t1) TOUCH(t2) GREEN(t3) CUBE
ITALK Year 3 Review Genoa, 21. June 2011
Training
• Neural Network trained with BPTT algorithm
• Training is made of sequences of words followed by the action of the robot (touch or not touch the object)
• Robot experiences both “correct” and “uncorrect” sentences:
Environment Language input Action
Red ball “TOUCH RED BALL”
Touch the ball
Green cube “TOUCH RED CUBE”
Not touch the ball
ITALK Year 3 Review Genoa, 21. June 2011
EXPERIMENT IIWord order
Environment Language input Expected action
Red cube “TOUCH GREEN” Not touch the cube
Red cube “TOUCH GREEN CUBE”
Not touch the cube
Green cube “TOUCH GREEN” Touch the cube
Green ball “TOUCH GREEN BALL” Touch the ball
Example of training sentences providing word order info:
After the training the robot was tested with additional utterances not originally included in the training set.
RESULTS:The robot is able to perform the related action and
correctly generalises to novel sentences
ITALK Year 3 Review Genoa, 21. June 2011
• Internal representations change their structureaccording to the word sequence experienced in input
Prediction: t2: {COLOUR} input Prediction: t3: {OBJECT} input
Ball
Cylinder
Cube
ITALK Year 3 Review Genoa, 21. June 2011
EXPERIMENT IIWord order
EXPERIMENT IIINew colour
• Generalisation to a new linguistic category was tested
• A new colour adjectives(blue) was included in the vocabulary
• The robot was tested withoutadditional training onlanguage (blue colour wasexperienced by the robot)
RESULTS: successful generalisation to new colour adjective
ITALK Year 3 Review Genoa, 21. June 2011
• the new colour adjective was successfullyclassified as other colour termsthe unseen input was correctly classified only on the basis of the word
orderthe distributional cue from the word order was thus successfully
interpreted as a clue to the semantic category (colour)
Self-organised Internal representation
ITALK Year 3 Review Genoa, 21. June 2011
EXPERIMENT IIINew colour
4.5 Origins of compositionality
University of Plymouth
ITALK Year 3 Review Genoa, 21. June 2011
Hypothesis
Exposing the robots to a task and an environment that present compositional features, such as
Move A Move BGreen ball Red ball Red cube
could lead the evolutionary process to organise the robot’s knowledge in a compositional structure.
This structure, in turn, could guide the communication system toward the evolution of compositional features.
ITALK Year 3 Review Genoa, 21. June 2011
Precondition
• Communication is grounded in agents’ sensorimotor coordination.
• Interaction between two robots• Collective task
ITALK Year 3 Review Genoa, 21. June 2011
Methodology
Evolutionary robotics
Neural network as control system Genetic algorithm to
evolve synaptic weights
Fittest reproduce
ITALK Year 3 Review Genoa, 21. June 2011
Experimental scenario
3 objects 3 pre-evolved primitive behaviours (touch, grasp, lift)
Communication channels allow robots to exchange signals
Robots are evolved to perform the same action on the same object
A
B
ITALK Year 3 Review Genoa, 21. June 2011
Task 4.4 Constructional grounding
University of Hertfordshire
ITALK Year 3 Review Genoa, 21. June 2011
• Hypotheses: • Children first learn object words (e.g. nouns) and then predicate words
(e.g. verbs) • Object bias: the first phonological words learnt assumed to be of ‘entity’
type• we assume prosodically salient words get associated with objects
• Approach: Semantic bootstrapping idea:• Classify lexicon into predicates and entities, and assign syntactic
constraints• Iterate hypothesis and revise lexicon• Models two-word stage (18-24 months)
o Results: Simulation o Using corpus from Kaspar studyo Result: approx. 65% of the words correctly classified
o Ongoing: interactive learning in the iCub studies with robot producing two word utterances
Grammar induction with simple semantic types
ITALK Year 3 Review Genoa, 21. June 2011
Review Meeting 2 St.Albans, UK
Semantics-Based Grammar Induction: Example
Lexicon state (before):
• Now facing an utterance, e.g. “You see that red big star, Kaspar”
↑ ↑ ↑phonologically known also referentially known
• Classify the words into two types: predicates and entitieso The learner already knows [star] belongs to the arg set, so
possible partitions are (predicates/entities):{ you } / { see, star } (wrong); { you, see } / { star } (wrong); { see } / { you, star } (correct)
• Register the results in lexicon: suppose the second (wrong) hypothesis is taken
Lexicon state (after):
ITALK Year 3 Review Genoa, 21. June 2011
Task 4.4 Constructional grounding
University of Southern Denmark
ITALK Year 3 Review Genoa, 21. June 2011
Two types of language grounding
• Experiential grounding
– Linking ‘elementary symbols’ to the world– aka ‘direct grounding’
• Higher-level symbol grounding
– Linking symbols to symbols (non-elementary to elementary)
– aka ‘grounding transfer’ (Pezzulo et al. 2011)
ITALK Year 3 Review Genoa, 21. June 2011
Two dimensions of transfer
• Learning new forms through tutoring
– Tutor (says): ‘An X is a Y that is Z’(Harnad 1990; Cangelosi and Riga 2006)
– Learning new symbols from known symbols:associate known meaning with new form
• Learning new meanings through reanalysis
– Learner (reasons): ‘X2 is a special type of X1’(Johnson 1999)
– Learning new uses of a symbol from known uses of the same symbol: associate known form with new meaning
ITALK Year 3 Review Genoa, 21. June 2011
Constructional grounding theory
• Johnson (1999)
– “a sign that is relatively easy for children to learn (the sourceconstruction) serves as the model for another more difficultsign (the target construction), because it occurs in contextsin which it exemplifies important properties of that sign in a way that is especially accessible to children”(Johnson 1999: 1)
ITALK Year 3 Review Genoa, 21. June 2011
‘Accessibility’
• Two potentially conflicting determinants:
– Quantitative salience: input frequency
Prediction: developmentally basic variants of a construction are acquired first because
they are (more) common in the input
– Qualitative salience: semantic concreteness
Prediction: developmentally basic variants of a construction are acquired first because
they are semantically (more) concrete
ITALK Year 3 Review Genoa, 21. June 2011
Case study: possession
• Prenominal possessives: [NP‘s NP]
– Developmentally basic: one of the earliest relation-ships marked in emerging child multiword speech(Lieven, Salomo & Tomasello 2009)
– Polyfunctional: wide semantic space (many different „meanings“)
– Semantically asymmetric: some meanings directlygrounded, others considerably more abstract
– Potential for reanalysis: Interpretive overlap betweenvariants
ITALK Year 3 Review Genoa, 21. June 2011
Semantic space
• Some relations marked bypossessives...
– Privileged access John’s car– Partonomy John’s hand– Kinship relations John’s brother– Property ascription John’s impatience– Creation John’s poem– Disposal John’s train (= the one he‘s on)– Participant–event John’s arrival– Location England’s cathedrals– Temporal setting–event Last year’s meeting– ...
ITALK Year 3 Review Genoa, 21. June 2011
concrete
abstract
Study overview
• Aims: For each type of possessive relation,
– What is the order of acquisition of relevant variants?– Which operationalisation of ‘accessibility’ is the more
powerful predictor of acquisition order?
• Data: 3 longitudinal CHILDES corpora– Brown corpus; target child: Sarah; age range: 2;3-5;1– Kuczaj corpus; target child: Abe; age range:2;4-5;0– Sachs corpus; target child: Naomi; age range: 1;2-5;1
• Method: Rank order correlations (Kendall)– Acquisition order vs. concreteness/frequency ranks
ITALK Year 3 Review Genoa, 21. June 2011
Data
• Overview:
– Automatic extraction (all N+N sequences in entire corpus)
– Plus 10 lines anterior and 5 lines posterior context
– Full manual coding in context
– Coded for: type & subtype of relation (concreteness rank)
Child N+N utterances coded Hits (caretaker) Hits (child) Total hitsSarah (Brown) 3675 399 101 500Abe (Kuczaj) 4509 90 175 265
Naomi (Sachs) 1674 180 141 321
ITALK Year 3 Review Genoa, 21. June 2011
Sample results: Abe
• Inherent possessives: age vs. concreteness
Kendall’s τ =.40, p=.53
Category Example Concreteness AcquiredPERSON – BODY PART Dad’s nose 1 3ANIMATE – BODY PART the skunk’s tail 2 2
PERSON – ATTRIBUTE/EXPERIENCE Mummy's name 3 1PERSON – EVENT my uncle’s funeral 4 4
SETTING/MEASURE – EVENT last year’s State Fair 5 5
ITALK Year 3 Review Genoa, 21. June 2011
Results
• For 2 of the 3 learners (Abe & Naomi),
– Sig. Cor: Input frequency with order of acquisition: the most common uses of the construction were learnedfirst
– Sig Cor: Input and output frequency
• For the third learner (Sarah),– Neither input frequency nor concreteness correlated with
order of acquisition
⇒Concreteness alone failed to yield a significant effect⇒Neither factor could fully explain the observed
orders alone
ITALK Year 3 Review Genoa, 21. June 2011
Conclusion
• ‘Quantitative salience’ outperforms ‘qualitative salience’ as a predictor of ease of acquisition (for our data)
multi-factorial model necessary to explain order of acquisition
Outlook• Corroboration by results from denser corpora/other
constructions desirable
• Computational implementation (i.e. grounded robotic learning of prenominal possessive cxns) currently underway (Zeschel& Tuci to appear)
ITALK Year 3 Review Genoa, 21. June 2011
WP 4 Summary Y3• 4.1 Language learning with MTRNN:
– Generalisation wrt noise, syntactic category, sentence complexity
• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-grained
synchrony– Tentative results on sync between syntax and motion
• 4.3 Word order facilitates– Learning syntactic categories and related actions– And to generalise
• 4.5 Origins of compositionality– Action compositionality to learn language compositionality in
interaction
• 4.4 Construction learning– Grammar induction– Higher level grounding in infants: Semantic salience (internal
reasoning) and input frequency (tutor input)
ITALK Year 3 Review Genoa, 21. June 2011
Evolutionary OriginsOf Compositionality 4.5
ActionActionHierarchy 4.1
AcousticPackages 4.2
Summary
Speech
GrammaticalConstructions
4.3
4.4
Lexicon Construction
ITALK Year 3 Review Genoa, 21. June 2011
ITALK Year 1 Review Düsseldorf, 1 July 2009
:Recognition of incorrect sentences
Evaluate auto-correction capability
Walk slowly.
TargetWals slowly.
incorrectnoise
比較 Init Vector
…
…
…
…
…
…
Error backpropagation
Walk slowly.
Generated…
…
…
…
…
…
Forward Calculation
Noise addition probability and Success Rate
50.0
55.0
60.0
65.0
70.0
75.0
80.0
85.0
90.0
95.0
100.0
0 5 10 20 30 40
CLEAN
NOISE
(%)
Acc
urac
y R
ate
(%)
ρ (Noise addition
MTRNN Activation pattern (example)
Cs
Cf
IO
Step
p u n c h □ t h e □ s m a l l □ y e l l o w □ b o x □ s l o w l y .
-2-1.5-1-0.5-0.5
0
0.5
1
PCA2PCA1
PCA3
punch the
yellow
slowlybo
x
-1-0.500.5
Punch the yellow box slowly.
-2-1.5-1-0.5-0.5
0
0.5
1
PCA2PCA1
Kick a small yellow ball.
PCA3
kick a
small
yellow
ball
-1-0.500.5
Cs: Transition of Slow Context Activation
Semantics-Based Grammar Induction: Results so far
• Crucial question is whether type hypotheses convergeo Simulation with the past experimental data (Year 1) suggest they doo Type mapping results: object bias improves learning with t-test significance
at p=.0035
• We are now testing it in an HRI experiment, with iCub actually responding to the partitipants with two-word utterances
F-score (P: precision, R: recall)
Baseline (no bias)Semantic bootstrapping with object bias
.4675 (P:.4502, R:.4863)
.6417 (P:.6190, R:.6663)
ITALK Year 3 Review Genoa, 21. June 2011
Overview• 4.1 Language learning with MTRNN:
– Generalisation wrt noise, syntactic category, sentencecomplexity
• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-
grained synchrony– Tentative results on sync between syntax and motion
• 4.3 Word order facilitates– Learning syntactic categories and related actions– And to generalise
• 4.4 Grammar Induction– Incremental lexicon development
• 4.5 Origins of compositionality– Action compositionality to learn language
compositionality in interaction
InternalReasoning
Input & InternalReasoning
InternalReasoning
Input &InternalReasoning
ITALK Year 3 Review Genoa, 21. June 2011
Motivation –Action Segmentation in Infants
• Children need to discover meaningful action units– Language helps to divide a
sequence of events into units– Prerequisite: synchrony between
language and events
• Described as acoustic packaging (AP) [Hirsh-Pasek and Golinkoff 1996]
• AP can provide a bottom-up action segmentation
ITALK Year 3 Review Genoa, 21. June 2011
Acoustic Package
Acoustic Package
t
t
Speech
Motion
A Computational Model ofAcoustic Packaging
• Segmentation of input cues– Acoustic temporal segmentation– Visual temporal segmentation
• Cue fusion– Temporal association of multi-
modal input streams– Results of the association process
are Acoustic Packages
/a:/ /p/ /h/ /2:/[noise1] /d/ /a/ /n/
Collection: Syllables From Multiple Runs
ITALK Year 3 Review Genoa, 21. June 2011
4.2 Summary
Summary
• Acoustic packaging provides a bottom-up action segmentation• Acoustic packaging makes use of interaction between modalities at an
early level• Acoustic packages simplify access to corresponding multimodal events at
a time• Synchrony between action and syntactic structure
Long term goals• Understanding actions• Language learning
Next Targets
• Consolidation of acoustic packages e.g. clustering, selection of relevant packages
• More complex Feedback
ITALK Year 3 Review Genoa, 21. June 2011
Slide Number 1OverviewProgress in WP4 in Y3Objectives & GoalsObjectives & Goals4.1 Generalization as a basis for the emergence symbolic systemObjectiveMTRNN modelExample for training Experimental procedureCs: Initial State AnalysisSummary4.2 Acoustic PackagingA Computational Model of�Acoustic PackagingGoalsAdditional Cues and their Role in Acoustic PackagingDetecting Moving Colored ObjectsAcoustic ProminenceFeedback based on �Acoustic PackagesProminent Syllable – Trajectory Color Association (en)iCub‘s Perspective�Synchrony between �verbal utterances and action Synchrony between �verbal utterances and action (USD, BIEL) 4.3 From single word lexicons to compositional languages Word OrderWord Order LearningRecurrent Neural NetworkExperiment I: SynonymsInput for Word Order LearningTrainingEXPERIMENT II� Word orderEXPERIMENT II� Word orderEXPERIMENT III�New colourEXPERIMENT III�New colour4.5 Origins of compositionality HypothesisPreconditionMethodologyExperimental scenarioSlide Number 40Slide Number 41Semantics-Based Grammar Induction: ExampleSlide Number 43Two types of language groundingTwo dimensions of transferConstructional grounding theory‘Accessibility’Case study: possessionSemantic spaceStudy overviewDataSample results: AbeResultsConclusion WP 4 Summary Y3SummarySlide Number 57:Recognition of incorrect sentencesNoise addition probability and Success RateSlide Number 60Cs: Transition of Slow Context Activation Semantics-Based Grammar Induction: Results so farOverviewMotivation – �Action Segmentation in InfantsA Computational Model of�Acoustic PackagingCollection: Syllables From Multiple Runs4.2 Summary