1 David Chen & Raymond Mooney Department of Computer Sciences University of Texas at Austin...

David Chen & Raymond MooneyDepartment of Computer Sciences

University of Texas at Austin

Learning to Sportscast: A Test of Grounded Language Acquisition

Motivation

• Constructing annotated corpora for language learning is difficult

• Children acquire language through exposure to linguistic input in the context of a rich, relevant, perceptual environment

• Learn to ground the semantics of language

• Learn language through correlated linguistic and visual inputs

Challenge

A linguistic input may correspond to many possible events

Block ?

Overview

• Sportscasting task

• Tactical generation

• Strategic generation

• Human evaluation

Learning to Sportscast

• Robocup Simulation League games

• No speech recognition– Record commentaries in text form

• No computer vision– Ruled-based system to automatically extract

game events in symbolic form

• Concentrate on linguistic issues

Robocup Simulation League

Pink4’s pass was intercepted by Purple6

Learning to Sportscast

• Learn to sportscast by observing sample human sportscasts

• Build a function that maps between natural language (NL) and meaning representation (MR)– NL: Textual commentaries about the game– MR: Predicate logic formulas that represent

events in the game

Mapping between NL/MR

NL: “Purple3 passes the ball to Purple5”

MR: Pass ( Purple3, Purple5 )

Semantic Parsing (NL MR)

Tactical Generation (MR NL)

Robocup Sportscaster TraceNatural Language Commentary Meaning Representation

Purple goalie turns the ball over to Pink8

badPass ( Purple1, Pink8 )

Pink11 looks around for a teammate

Pink8 passes the ball to Pink11

Purple team is very sloppy today

Pink11 makes a long pass to Pink8

Pink8 passes back to Pink11

turnover ( Purple1, Pink8 )

pass ( Pink11, Pink8 )

ballstopped

kick ( Pink11 )

kick ( Pink8)

kick ( Pink11 )

kick ( Pink8 )

ballstopped

kick ( Pink11 )

kick ( Pink8)

kick ( Pink11 )

kick ( Pink8 )

ballstopped

kick ( Pink11 )

kick ( Pink8)

kick ( Pink11 )

kick ( Pink8 )

P6 ( C1, C19 )

P5 ( C1, C19 )

P2 ( C22, C19 )

P2 ( C19, C22 )

P1 ( C22 )

P1( C19 )

P1 ( C22 )

P1 ( C19 )

Robocup Data

• Collected human textual commentary for the 4 Robocup championship games from 2001-2004.– Avg # events/game = 2,613– Avg # sentences/game = 509

• Each sentence matched to all events within previous 5 seconds.– Avg # MRs/sentence = 2.5 (min 1, max 12)

• Manually annotated with correct matchings of sentences to MRs (for evaluation purposes only).

Overview

Tactical Generation

• Learn how to generate NL from MR

• Example:

• Two steps1. Disambiguate the training data

2. Learn a language generator

Pass(Pink2, Pink3) “Pink2 kicks the ball to Pink3”

System Overview

Purple7 loses the ball to Pink2

Sportscaster Robocup Simulator

Ambiguous Training Data

Pink2 kicks the ball to Pink5

Pink8 shoots the ball

Turnover ( purple7 , pink2 )

Pass ( pink5 , pink8)

Pass ( Purple5, Purple7 )

Kick ( pink2 )Pass ( pink2 , pink5 )Kick ( pink5 )

BallstoppedKick ( pink8 )

System Overview

Pass ( Purple5, Purple7 )

Semantic Parser Learner

Initial SemanticParser

System Overview

Pass ( purple5, purple7 )

Initial SemanticParser

Unambiguous Training Data

Pink2 kicks the ball to Pink5Pink5 makes a long pass

to Pink8Pink8 shoots the ball Kick ( pink8 )

Pass ( pink2 , pink5 )

Kick ( pink2 )

Kick ( pink5 )

System Overview

SemanticParser

Kick ( pink2 )

Kick ( pink5 )

System Overview

SemanticParser

Kick ( pink5 )

System Overview

SemanticParser

Kick ( pink5 )

System Overview

SemanticParser

Semantic Parser Learners

• Learn a function from NL to MR

NL: “Purple3 passes the ball to Purple5”

MR: Pass ( Purple3, Purple5 )

Semantic Parsing (NL MR)

Tactical Generation (MR NL)

•We experiment with two semantic parser learners–WASP (Wong & Mooney, 2006; 2007)–KRISP (Kate & Mooney, 2006)

WASP: Word Alignment-based Semantic Parsing

• Uses statistical machine translation techniques– Synchronous context-free grammars (SCFG)

(Wu, 1997; Melamed, 2004; Chiang, 2005)– Word alignments (Brown et al., 1993; Och &

Ney, 2003)

• Capable of both semantic parsing and tactical generation

KRISP: Kernel-based Robust Interpretation by Semantic Parsing

• Productions of MR language are treated like semantic concepts

• SVM classifier is trained for each production with string subsequence kernel

• These classifiers are used to compositionally build MRs of the sentences

• More resistant to noisy supervision but incapable of tactical generation

Matching

• Ability to find correct NL/MR pair• 4 Robocup championship games from 2001-2004.

– Avg # events/game = 2,613– Avg # sentences/game = 509

• Leave-one-game-out cross-validation• Metric:

– Precision: % of system’s annotations that are correct– Recall: % of gold-standard annotations correctly

produced– F-measure: Harmonic mean of precision and recall

Systems

Learner

KRISPER

(Kate & Mooney, 2007)

WASPER WASP

WASPER-GEN WASP’s language generator

KRISPER and WASPER

SemanticParser

Kick ( pink5 )

(KRISP/WASP)

Systems

Learner

KRISPER

WASPER WASP

WASPER-GEN

TacticalGenerator

Kick ( pink5 )

Tactical Generator Learner(WASP)

Matching Results

Average results on leave-one-game-out cross-validation

re randomKRISPERWASPERWASPER-GEN

Overview

Strategic Generation

• Generation requires not only knowing how to say something (tactical generation) but also what to say (strategic generation).

• For automated sportscasting, one must be able to effectively choose which events to describe.

Example of Strategic Generation

pass ( purple7 , purple6 )

ballstopped

kick ( purple6 )

ballstopped

kick ( purple2 )

kick ( purple3 )

badPass ( purple3 , pink9 )

turnover ( purple3 , pink9 )

Example of Strategic Generation

ballstopped

kick ( purple6 )

ballstopped

kick ( purple2 )

kick ( purple3 )

badPass ( purple3 , pink9 )

Strategic Generation

• For each event type (e.g. pass, kick) estimate the probability that it is described by the sportscaster.

• Requires correct NL/MR matching– Use estimated matching from tactical

generation– Iterative Generation Strategy Learning

Iterative Generation Strategy Learning (IGSL)

• Directly estimates the likelihood of an event being commented on

• Self-training iterations to improve estimates

• Uses events not associated with any NL as negative evidence

Strategic Generation Performance

• Evaluate how well the system can predict which events a human comments on

• Metric:– Precision: % of system’s annotations that are correct

– Recall: % of gold-standard annotations correctly produced

– F-measure: Harmonic mean of precision and recall

Strategic Generation Results

Average results on leave-one-game-out cross-validation

inferred fromWASPinferred fromKRISPERinferred fromWASPERinferred fromWASPER-GENIGSL

inferred fromgold matching

Overview

• 4 fluent English speakers as judges• 8 commented game clips

– 2 minute clips randomly selected from each of the 4 games

– Each clip commented once by a human, and once by the machine

• Presented in random counter-balanced order• Judges were not told which ones were human or

machine generated

Human Evaluation (Quasi Turing Test)

Demo Clip

• Game clip commentated using WASPER-GEN with IGSL, since this gave the best results for generation.

• FreeTTS was used to synthesize speech from textual output.

Human Evaluation

ScoreEnglish Fluency

Semantic Correctness

Sportscasting Ability

5 Flawless Always Excellent

4 Good Usually Good

3 Non-native Sometimes Average

2 Disfluent Rarely Bad

1 Gibberish Never Terrible

Human Evaluation

CommentatorEnglishFluency

SportscastingAbility

Human 3.94 4.25 3.63

Machine 3.44 3.56 2.94

Difference 0.5 0.69 0.69

ScoreEnglish Fluency

Sportscasting Ability

5 Flawless Always Excellent

4 Good Usually Good

3 Non-native Sometimes Average

2 Disfluent Rarely Bad

1 Gibberish Never Terrible

Future Work

• Expand MRs to beyond simple logic formulas• Apply approach to learning situated language in a

computer video-game environment (Gorniak & Roy, 2005)

• Apply approach to captioned images or video using computer vision to extract objects, relations, and events from real perceptual data (Fleischman & Roy, 2007)

Conclusion

• Current language learning work uses expensive, unrealistic training data.

• We have developed a language learning system that can learn from language paired with an ambiguous perceptual environment.

• We have evaluated it on the task of learning to sportscast simulated Robocup games.

• The system learns to sportscast almost as well as humans.

Backup Slides

Systems

Disambiguation Learning language generator

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

KRISPER-WASP KRISP WASP

WASP with gold matching

N/A WASP

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

Systems

KRISPER and WASPER

SemanticParser

Kick ( pink5 )

(KRISP/WASP)

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

Systems

WASPER-GEN

TacticalGenerator

Kick ( pink5 )

Tactical Generator Learner(WASP)

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

Systems

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

Systems

Matching

• 4 Robocup championship games from 2001-2004.– Avg # events/game = 2,613

– Avg # sentences/game = 509

• Leave-one-game-out cross-validation• Metric:

– Precision: % of system’s annotations that are correct

– Recall: % of gold-standard annotations correctly produced

– F-measure: Harmonic mean of precision and recall

Matching Results

Average results on leave-one-outexperiments

re WASPKRISPERWASPERWASPER-GEN

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

Systems

WASP Random WASP

KRISPER

KRISP N/A

WASPER WASP WASP

N/A WASP

Lower baseline

Upper baseline

SystemsTactical

Generation

Tactical Generation

• 4 Robocup championship games from 2001-2004.– Avg # events/game = 2,613– Avg # sentences/game = 509

• Leave-one-game-out cross-validation• NIST score

– Evaluate the quality of machine translations based on matching n-grams

– BLEU metric with some modifications– More weight for rarer n-grams– Less sensitive to translation length

Tactical Generation Results

Average results on leave-one-out experiments

WASPER

KRISPER-WASPWASPER-GEN

WASP with goldmatching

Parsing Results

010203040

5060708090

Average results on leave-one-out experiments

KRISPER

WASPER

KRISPER-WASPWASPER-GEN

WASP with goldmatching

Matching Results

Parse Results

Generation Results

Strategic Generation Results

Grounded Language Learning in Robocup Robocup Simulator

Sportscaster

Simulated Perception

Perceived Facts

Score!!!!Grounded

Language LearnerLanguageGenerator

SemanticParser

SCFG Score!!!!

“Mary is on the phone”

“Mary is on the phone”???

Ironing(Mommy, Shirt)

Working(Sister, Computer)

Carrying(Daddy, Bag)

Talking(Mary, Phone)

Sitting(Mary, Chair)

Grounding Language

“Spanish goalkeeper Iker Casillas blocks the ball”

Grounding Language

“Spanish goalkeeper Iker Casillas blocks the ball”

Grounding Language

Blocks

• WordNet - (v) parry, block, deflect (impede the movement of (an opponent or a ball)) "block an attack“

• Merriam-Webster - intransitive verb: to block an opponent in sports

Grounding Language

Blocks

• WordNet - (v) parry, block, deflect (impede the movement of (an opponent or a ball)) "block an attack“

• Merriam-Webster - intransitive verb: to block an opponent in sports

Grounding Language

Blocks

Robocup Sportscaster Trace

purple6 passes to purple2

purple3 loses the ball to pink9

purple2 makes a short pass to purple3

ballstopped

kick ( purple6 )

kick ( purple2 )

kick ( purple3 )

Natural Language Commentary Meaning Representation

kick ( purple6 )

kick ( purple2 )

kick ( purple3 )

kick ( purple 3 )

ballstopped

kick ( purple6 )

kick ( purple2 )

kick ( purple3 )

kick (purple 3

kick ( purple 3 )

kick ( purple6 )

kick ( purple2 )

kick ( purple3 )

kick (purple 3

kick ( purple 3 )

kick ( purple6 )

kick ( purple2 )

kick ( purple3 )

kick (purple 3 )

Negative Evidence

1 David Chen & Raymond Mooney Department of Computer Sciences University of Texas at Austin...

Documents

PrasadL11Classify1 Vector Space Classification Adapted from Lectures by Raymond Mooney and Barbara Rosario

11 CS 388: Natural Language Processing: Discriminative Training and Conditional Random Fields (CRFs) for Sequence Labeling Raymond J. Mooney University

Watch Listen & Learn: Co- training on Captioned Images and Videos Sonal Gupta, Joohyun Kim, Kirstenn Garuman and Raymond Mooney

MILWAUKEE PUBLIC SCHOOLS Monthly Planning Template Mary Mooney Tara Raymond February 8, 2012

David Chen Advisor: Raymond Mooney Research Preparation Exam August 21, 2008 Learning to Sportscast: A Test of Grounded Language Acquisition

Relational Learning of Pattern-Match Rules for Information Extraction Mary Elaine Califf Raymond J. Mooney

1 Sequence Labeling Raymond J. Mooney University of Texas at Austin

Razvan C. Bunescu Raymond J. Mooney

Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney -Pratiksha Shah

1 CS 343: Artificial Intelligence Probabilistic Reasoning and Naïve Bayes Raymond J. Mooney University of Texas at Austin

11 CS 388: Natural Language Processing: Syntactic Parsing Raymond J. Mooney University of Texas at Austin

1 CS 391L: Machine Learning Text Categorization Raymond J. Mooney University of Texas at Austin

1 CS 343: Artificial Intelligence Natural Language Processing Raymond J. Mooney University of Texas at Austin

1 1 CS 388: Natural Language Processing: Statistical Parsing Raymond J. Mooney University of Texas at Austin

PrasadL14VectorClassify1 Vector Space Text Classification Adapted from Lectures by Raymond Mooney and Barbara Rosario

1 CS 343: Artificial Intelligence Neural Networks Raymond J. Mooney University of Texas at Austin

1 CS 391L: Machine Learning: Computational Learning Theory Raymond J. Mooney University of Texas at Austin

1 CS 391L: Machine Learning: Experimental Evaluation Raymond J. Mooney University of Texas at Austin

Adbuctive Markov Logic for Plan Recognition Parag Singla & Raymond J. Mooney Dept. of Computer Science University of Texas, Austin

1 CS 388: Natural Language Processing Introduction Raymond J. Mooney University of Texas at Austin