1

CSC 594 Topics in AI –Applied Natural Language Processing

Fall 2009/2010

1. Introduction

2

What is NLP?What is NLP? Natural Language Processing (NLP) is a field Natural Language Processing (NLP) is a field

in Artificial Intelligence (AI) devoted to in Artificial Intelligence (AI) devoted to creating computers that use natural creating computers that use natural language as input and/or output. language as input and/or output.

3

NLP involves other disciplines..NLP involves other disciplines..

LinguisticsLinguistics– NLP is also calledNLP is also called ”Computational ”Computational

Linguistics”Linguistics” PsychologyPsychology Mathematics and StatisticsMathematics and Statistics Information TheoryInformation Theory Computer ScienceComputer Science

4

Machines that Can SpeakMachines that Can Speak

HAL 9000 in “2001: A Space Odyssey”HAL 9000 in “2001: A Space Odyssey”

5

C3PO C3PO in Star Warsin Star Wars

Machines that Can Speak (cont.)Machines that Can Speak (cont.) KITT KITT

in Knight Riderin Knight Rider

6

But Still a Sci-Fi…But Still a Sci-Fi…

““However, as 2001 approached it became However, as 2001 approached it became clear that clear that 2001'2001's predictions in computer s predictions in computer technology were far fetched. Natural technology were far fetched. Natural language, lip reading, planning and plain language, lip reading, planning and plain common sense in computers were still the common sense in computers were still the stuff of science fiction.”stuff of science fiction.”HAL 9000 - WikipediaHAL 9000 - Wikipedia

7

NLP is AI-completeNLP is AI-complete

• ““The most difficult problems in AI manifest The most difficult problems in AI manifest themselves in human language phenomena.”themselves in human language phenomena.”

• Use of language is the touchstone of intelligent Use of language is the touchstone of intelligent behavior.behavior.

8

Test for Intelligence – Turing TestTest for Intelligence – Turing Test Alan Turing (1950) proposed a Alan Turing (1950) proposed a

test of a machine's capability to test of a machine's capability to perform human-like perform human-like conversation.conversation.

A human judge engages in a natural language A human judge engages in a natural language conversation with two other parties, one a conversation with two other parties, one a human and the other a machine; if the judge human and the other a machine; if the judge cannot reliably tell which is which, then the cannot reliably tell which is which, then the machine is said to pass the test. machine is said to pass the test.

9

Early Conversational ProgramsEarly Conversational Programs

• ELIZA (by Joseph Weizenbaum)(by Joseph Weizenbaum), 1966, 1966– A psychotherapistA psychotherapist– No real understanding; simple pattern-matching No real understanding; simple pattern-matching

to respond to user input -> canned responsesto respond to user input -> canned responses

(my ?x depresses me) (why does your ?x depress you) 5

(life ?x) (why do you say it ?x) 3

(I could ?x) (you could ?x) 2

(because ?x) (that is a good reason) 3

(?x) (tell me more) 0

10

>> Hello.How do you do. Please state your problem.>> Men are all alike.In what way ?>> They're alwas bugging us about something or other.I'm not sure I understand you fully.>> They're always bugging us about something or other.Can you think of a specific example ?>> Well, my boyfriend made me come here.Your boyfriend made you come here ?>> He says I'm depressed much of the time.I am sorry to hear that you are depressed.

11

Recent Conversational ProgramsRecent Conversational ProgramsLoebner Prize &Loebner Prize &

Chatterbots/Chatbots (1)Chatterbots/Chatbots (1)• In 1990, Hugh Loebner started an annual Turing In 1990, Hugh Loebner started an annual Turing

Test competition.Test competition.• Conversational Programs are nowadays called Conversational Programs are nowadays called

ChatterbotsChatterbots (or (or ChatbotsChatbots).).• $100,000 to the first bot that judges cannot $100,000 to the first bot that judges cannot

distinguish from a real human in a Turing test that distinguish from a real human in a Turing test that includes text, visual, and auditory input.includes text, visual, and auditory input.

• The Prize dissolves once the $100,000 prize is The Prize dissolves once the $100,000 prize is won.won.

12

Loebner Prize &Loebner Prize &Chatterbots/Chatbots (2)Chatterbots/Chatbots (2)

• Nobody has won the prize yet.Nobody has won the prize yet.

13

Real-World Applications of NLP

• NLP can be stand-along applications or NLP can be stand-along applications or components embedded in other systems.components embedded in other systems.

• Major NLP applications include:Major NLP applications include:– Machine translation – Question answering– Summarization – Conversational agents

14

Commercial World• Lot’s of exciting stuff going on…

Powerset

Source: Jurafsky & Martin “Speech and Language Processing”

15

1. Machine Translation (MT)1. Machine Translation (MT)

• One of the very earliest pursuits in computer One of the very earliest pursuits in computer science (after WWII).science (after WWII).

• Basic approaches:Basic approaches:– Inter-lingualInter-lingual (rule-based) (rule-based)– Direct translationDirect translation (corpus-based) (corpus-based) more more

popular these dayspopular these days• Example: Example: Google Translate

– MT engine (based on SYSTRAN system developed in EC).

16

2. Question Answering2. Question Answering

• Finds an answer (not a document) to a question Finds an answer (not a document) to a question typed in as a natural language sentence (not typed in as a natural language sentence (not keywords).keywords).

• Most systems can only answer simple, trivial pursuit Most systems can only answer simple, trivial pursuit type questions.type questions.

• Example: Example: Ask.com

• Some search engines perform limited, phrase-based Some search engines perform limited, phrase-based Q&A, e.g. Q&A, e.g. Google

17

3. Text Summarization3. Text Summarization

• Create a summary of a text or texts.Create a summary of a text or texts.• Many difficult problems, including:Many difficult problems, including:

– ParaphrasesParaphrases– Anaphora (e.g.“it”, “they”)Anaphora (e.g.“it”, “they”)

18

4. Analyzing Web Documents4. Analyzing Web Documents

• Recently there have been many NLP applications Recently there have been many NLP applications which analyze (not just retrieve) wwhich analyze (not just retrieve) weblogs, discussion forums, message boards, user groups, and other forms of user generated media– Product marketing information– Political opinion tracking– Social network analysis– Buzz analysis (what’s hot, what topics are people talking

about right now).

Source: Jurafsky & Martin “Speech and Language Processing”

19

5. NLP in IR• Query expansion

– Add synonyms, related words to the query terms to improve search results.

– Example: Google AdWords tool

• NOTE: Stemming is NOT a NLP technique!!!

20Source: Marti Hearst, i256, at UC Berkeley

21

NLP Tasks• Those NLP applications require several NLP analyses:

– Word tokenization– Sentence boundary detection– Part-of-speech (POS) tagging

• to identify the part-of-speech (e.g. noun, verb) of each word– Named Entity (NE) recognition

• to identify proper nouns (e.g. names of person, location, organization; domain terminologies)

– Parsing• to identify the syntactic structure of a sentence

– Semantic analysis• to derive the meaning of a sentence

22

Different Levels of Language Analysis

• Phonology– Speech audio signal to phonemes

• Morphology– Inflection (e.g. “I”, “my”, “me”; “eat”, “eats”, “ate”, “eaten”)– Derivation (e.g. “teach”, “teacher”, “nominate”, “nominee”)

• Syntax– Part-of-speech (noun, verb, adjective, preposition, etc.)– Phrase structure (e.g. noun phrase, verb phrase)

• Semantics– Meaning of a word (e.g. “book” as a bound volume or an

accounting ledger) or a sentence • Discourse

– Meaning and inter-relation between sentences

23

Why is NLP so hard..?Why is NLP so hard..?

• Understanding natural languages is hard … Understanding natural languages is hard … because of inherent because of inherent ambiguityambiguity

• Engineering NLP systems is also hard … Engineering NLP systems is also hard … because of:because of:– Huge amount of data resources needed (e.g. Huge amount of data resources needed (e.g.

grammar, dictionary, documents to extract grammar, dictionary, documents to extract statistics from)statistics from)

– Computational complexity (intractable) of Computational complexity (intractable) of analyzing a sentenceanalyzing a sentence

24

Ambiguity (1)Ambiguity (1)

““Get the cat with the gloves.”

Source: Marti Hearst, i256, at UC Berkeley

25

Ambiguity (2)Ambiguity (2)Find at least 5 meanings of this sentence:

“I made her duck”

1. I cooked waterfowl for her benefit (to eat)2. I cooked waterfowl belonging to her3. I created the (plaster?) duck she owns4. I caused her to quickly lower her head or body5. I waved my magic wand and turned her into

undifferentiated waterfowl

Source: Jurafsky & Martin “Speech and Language Processing”

26

Ambiguity (3)Ambiguity (3)

Some ambiguous headlinesSome ambiguous headlines

• Juvenile Court to Try Shooting Defendant• Teacher Strikes Idle Kids• Kids Make Nutritious Snacks• Bush Wins on Budget, but More Lies Ahead• Hospitals are Sued by 7 Foot Doctors

Source: Marti Hearst, i256, at UC Berkeley

27

Ambiguity is Pervasive

• Phonetics– I mate or duck– I’m eight or duck– Eye maid; her duck– Aye mate, her duck– I maid her duck– I’m aid her duck– I mate her duck– I’m ate her duck– I’m ate or duck– I mate or duck

Source: Jurafsky & Martin “Speech and Language Processing”

Sound like“I made her duck”

28

• Lexical category (part-of-speech)– “duck” as a noun or a verb

• Lexical Semantics (word meaning)– “duck” as an animal or a plaster duck statue

• Compound nouns– e.g. “dog food”, “Intelligent design scores …”

• Syntactic ambiguity

• [But semantics can sometimes help disambiguate]

“I saw a man on the hill with a telescope”

“I saw a man on the hill with a hat”

29

Dealing with AmbiguityDealing with AmbiguityFour possible approaches:

1. Formal approaches -- Tightly coupled interaction among processing levels; knowledge from other levels can help decide among choices at ambiguous levels.

2. Pipeline processing that ignores ambiguity as it occurs and hopes that other levels can eliminate incorrect structures.

3. Probabilistic approaches based on making the most likely choices

4. Don’t do anything, maybe it won’t matter

Source: Jurafsky & Martin “Speech and Language Processing”

30

The Bottom LineThe Bottom Line

• Complete NL Understanding (thus general Complete NL Understanding (thus general intelligence) is impossible.intelligence) is impossible.

• But we can make incremental progress.But we can make incremental progress.• Also we have made successes in limited domains.Also we have made successes in limited domains.• [But NLP is costly – Lots of work and resources are [But NLP is costly – Lots of work and resources are

needed, but the amount of return is sometimes not needed, but the amount of return is sometimes not worth it.]worth it.]

31

Statistical Approaches to NLP• Get large text collections (corpora)• Compute statistics over the words in those collections• Surprising results:

– Getting more data is better than fine-tuning algorithms!

Source: Marti Hearst, i256, at UC Berkeley

Banko & Brill ‘01

32

Sentence AnalysisSentence Analysis

(ACTION ingest

(ACTOR John-1)

(PATIENT food))

Syntactic structure Semantic structure

“John ate the cake”

S

NP V NP

“John” “ate” “the cake”

33

Syntactic ParsingSyntactic Parsing

GrammarR0:R1:R2:R3:R4:R5:R6:R7: cake"" N

the"" Det ate"" V

John"" NP V VG

NPVG VPN Det NP

VP NP S

S

NP VP

V NP

“John” “ate”

“the”

Det N

“cake”

The process of deriving the phrase structure The process of deriving the phrase structure of a sentence is called “parsing”.of a sentence is called “parsing”.

The structure (often represented by a The structure (often represented by a Context-Free parse tree) is based on the Context-Free parse tree) is based on the grammar.grammar.

34

Parsing AlgorithmsParsing Algorithms

• Top-down Parsing -- (top-down) derivationTop-down Parsing -- (top-down) derivation• Bottom-up ParsingBottom-up Parsing• Chart ParsingChart Parsing• Earley’s Algorithm – most efficient, Earley’s Algorithm – most efficient, O(nO(n33))• Left-corner Parsing – optimization of Earley’sLeft-corner Parsing – optimization of Earley’s• and lots more…and lots more…

35

(Bottom-up) Chart Parsing(Bottom-up) Chart Parsing“John ate the cake”0 1 2 3 4

John""NP,1,0

VP NPS,4,0

NPVGVP,2,1

(2) shift 2

(1) shift 1 “John”

(7) shift 2

the""Det,3,2 cake""N,4,3

NPVG VP,4,1

(9) reduce

(10) reduce

ate""V,2,1

(5) shift 2

VPNPS,1,0 VVG,2,1 NDetNP,3,2

NDet NP,4,2

(3) shift 1 “ate” (6) shift 1 “the” (8) shift 1 “cake”

(4) shift 2

(11) reduce

---

---

---

0 1 2 3 4

cake"" Nthe"" Det

ate"" VJohn"" NP V VG

NPVG VPN Det NP

VP NP S

Grammar

36

Earley’s AlgorithmEarley’s Algorithm

“John ate the cake”0 1 2 3 4

cake"" Nthe"" Det

ate"" VJohn"" NP V VG

NPVG VPN Det NP

VP NP S

Grammar

VP NPS,0,0 VPNPS,1,0 VP NPS,4,0

NPVG VP,1,1 NPVGVP,2,1

(2) scanner“John”

(4) predictor

(5) scanner“ate”

NDet NP,2,2

(7) predictor

NDetNP,3,2

(8) scanner“the”

NDet NP,4,2

(9) scanner“cake”

NPVG VP,4,1

(10) completor

(11) completor

SS',0,0

(1) predictor

VVG,1,1 VVG,2,1

(3) predictor

(6) completor

SS',4,0

(12) completor

37

Demo using my CF parser

38

Probabilistic ParsingProbabilistic Parsing

• For ambiguous sentences, we’d like to know which For ambiguous sentences, we’d like to know which parse tree is more likely than others.parse tree is more likely than others.

• So we must assign probability to each parse tree … So we must assign probability to each parse tree … but how?but how?

• A probability of a parse tree A probability of a parse tree t t isis where where rr is a rule used in is a rule used in tt..

and and p(r)p(r) is obtained from a (annotated) corpus. is obtained from a (annotated) corpus.

r

rptp )()(

39

Partial ParsingPartial Parsing

• Parsing fails when the coverage of the grammar is Parsing fails when the coverage of the grammar is not complete – but it’s almost impossible to write not complete – but it’s almost impossible to write out all legal syntax (without accepting out all legal syntax (without accepting ungrammatical sentences).ungrammatical sentences).

• We’d like to at least get pieces even when full We’d like to at least get pieces even when full parsing fails.parsing fails.

• Why not abandon full parsing and aim for partial Why not abandon full parsing and aim for partial parsing from the start…parsing from the start…

40

Semantic Analysis (1)Semantic Analysis (1) Derive the meaning of a sentence.Derive the meaning of a sentence. Often applied on the result of syntactic Often applied on the result of syntactic

analysis.analysis.““JohnJohn ateate the cakethe cake.”.” NP V NPNP V NP ((action((action INGEST) ; syntactic verb INGEST) ; syntactic verb (actor (actor JOHN-01) ; syntactic subjJOHN-01) ; syntactic subj (object (object FOOD)) ; syntactic objFOOD)) ; syntactic obj

To do semantic analysis, we need a (semantic) To do semantic analysis, we need a (semantic) dictionary (e.g. WordNet, dictionary (e.g. WordNet, http://www.cogsci.princeton.edu/~wn/).).

41

Semantic Analysis (2)Semantic Analysis (2)

Semantics is a double-edged sword…Semantics is a double-edged sword…– Can resolve syntactic ambiguityCan resolve syntactic ambiguity

““I saw a man on the hill with a I saw a man on the hill with a telescopetelescope”” ““I saw a man on the hill with a I saw a man on the hill with a hathat””

– But introduces semantic ambiguityBut introduces semantic ambiguity ““She walked towards the She walked towards the bank”bank”

But in human sentence processing, we seem But in human sentence processing, we seem to resolve both types of ambiguities to resolve both types of ambiguities simultaneously (and in linear time as well)…simultaneously (and in linear time as well)…

42

Demo using my Unification parser