When the Shoe Fits:

Cross-Situational Learning in

Realistic Learning Environments

Tamara N. Medina1

John Trueswell1

Jesse Snedeker2

Lila Gleitman1

1Institute for Research in Cognitive Science, University of Pennsylvania2Department of Psychology, Harvard University

1998, Third Edition, Completely Revised

• “Where are your shoes?”• “Oh, what dirty shoes!”• “Let’s take your shoes off.”• “I’ll put your shoes on.”• “Look what nice new shoes.”

“A baby hears a word like “shoes”, for example, over and over again in daily life as the one constant sound in a large variety of statements. In one day you may say to him:”“That word “shoes” is the one sound which occurs in all those sentences and it is always associated with those things that go on his feet. Eventually he will associate the spoken sound with the objects and when he has made that association, he will have learned what the word “shoes” means.”

Cross-Situational Learning

“Find a set of possible meanings in each situation and intersect those sets across all situations in which a word occurs to determine the meaning for that word.”

Siskind, J.M. (1996, Cognition)

“It’s not so easy!”– Augustine, Locke, Quine, Gleitman, Fodor, Siskind, etc.

Frame / Level of DescriptionAnimal? Dog? Terrier? Fido? Friendly?

Referential UncertaintyWhich referent?

Frame Problem Solved?• Xu & Tenenbaum (2007): learn appropriate extensions

of a word via Bayesian inference (note “suspicious coincidences”)

“VASH” “VASH”“VASH”

Reference Problem Solved?• Yu & Smith (2007): learn word-object associations in

spite of “referential uncertainty”

“DOON” … “VASH” … “MIPEN” … “ZANT”

“VASH” ??

“VASH” !

Goal: Explore cross-situational word learning using naturalistic settings: both the cluttered and potentially uninformative or misleading environments and these somewhat more transparent ones.

Overview

• Adaptation of the Human Simulation Paradigm (Gillette et al., 1999)

• Norming Study

• Current Study

• 2 measures to evaluate word learning

Adaptation of Human Simulation Paradigm (Gillette et al., 1999)

• Selected stimuli based on results of earlier norming study:Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In

progress). Machine implementation of a verb learning algorithm.

• Large video corpus of parent-child interactions in natural settings (home, outdoors, etc.):Snedeker, J. (2001). Interactions between infants (12-15 months)

and their parents in four settings. Unpublished corpus.

Norming Study

• Identified 48 most frequently occurring content words.• Randomly selected six instances of each word.• Each instance was edited into a 40-second “vignette”.• Sound turned off.

– Visual context only cue to word meaning, placing viewers in the situation of the early word learner.

• Utterance of target word indicated by a BEEP.

Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In progress). Machine implementation of a verb learning algorithm.

30 sec

10 sec

<BEEP>

(silence)

(silence)

(silence)

Drawings courtesy of Emily Trueswell

Subject Guesses(Target Word = Shoe)

Subject Guesses(Target Word = Horse)

…

Subject Guesses(Target Word = Shoe)

No opportunity for cross-situational learning in norming study

8% correct 83% correct 0% correct

High Informative(>50% correct)

Low Informative(<33% correct)

7% of Vignettes = High Informative (atypical)

90% of Vignettes = Low Informative (typical)

Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In progress). Machine implementation of a verb learning algorithm.

Questions

• Does the observation of multiple naturalistic learning instances generate a gradually increasing learning curve?

• With regard to informativeness, given only the Low Informative vignettes, is cross-situational learning successful? Or is a High Informative instance necessary?

• If learners are building an interpretation across instances, does it matter when the High Informative instance occurs?

Current Study

Similar to norming study,

BUT allows for cross-situational learning

30 sec

10 sec

(silence)

(silence)

(silence)

Current Study Allows for Cross-Situational Learning

“VASH”

“VASH”(Target Word = Shoe)

Subject makes guess

Subject ratesConfidence (1 to 5)

…

Opportunity for cross-situational learning

“MIPEN”(Target Word = Horse)

Subject makes guess

Subject ratesConfidence (1 to 5)

“VASH”(Target Word = Shoe)

Subject makes guess

Subject ratesConfidence (1 to 5)

Final Conjectures and Confidence Ratings for each word

Manipulated the distribution of informative events

• For each of 8 Target nouns, there were:– 1 “High Informative” vignette (>50% of participants correct in

norming study)– 4 “Low Informative” vignettes (<33% of participants correct in

norming study)

• 4 Filler nouns– 5 “Low Informative” vignettes

• Participants assigned to one of 4 orders:– High Informative First: H-L-L-L-L– High Informative Middle: L-L-H-L-L– High Informative Last: L-L-L-L-H– High Informative Absent: L-L-L-L-L

(fifth vignette is a repeat of the first)

Accuracy Across Vignettes

Accuracy Across Vignettes

Accuracy Across Vignettes

Accuracy Across Vignettes

Accuracy Across Vignettes

Interim Summary: What have we learned about learning?

1. Gradual learning from partially informative instances is small to nonexistent.

2. Successful learning depends on the presence of a High Informative instance. (Epiphany!)

3. Low Informative instances have a corrupting influence on later-occurring High Informative instances. (Cross-situational learning of the bad sort.)

Epiphany!

Successful learning depends on the presence of a High Informative instance.

• Explicit and immediate insight?

• After using evidence from later instances?– High Informative instance provides key for

interpreting later instances.

Confidence on Correct Guesses across Vignettes

Confidence on Correct Guesses across Vignettes

Confidence on Correct Guesses across Vignettes

Confidence on Correct Guesses across Vignettes

Implications

• Shape of the word learning curve may be very different than what cross-situational learning models (e.g., Yu & Smith, 2007) have suggested:

Rapid

Incremental

Implications

• Successful word learning from cross-situational observation requires the occurrence of a highly informative instance.– But must it occur first?

• Greater delay between instances of a novel word: Every day is a new day.

• Multiple High Informative learning instances.– Previous studies which show striking rapid word learning

are such cases.

• Less weight on interpretations of Low Informative instances.

Logically, no!

Implications

• A High Informative instance is the first step in successful cross-situational word learning.– Prior Low Informative instances might not be

remembered over time.– Later Low Informative instances become

useful (confirmatory evidence?)• Supported by rising confidence levels after a High

Informative vignette.

Cross-situational learning does work,but only when the shoe fits.

Accuracy Confidence

Perseverance of First Guess