1 The Long Road from Text to Meaning Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex

1

The Long Road from Text to Meaning

Adam KilgarriffLexical Computing LtdLexicography MasterClass LtdUniversities of Leeds and Sussex

Adam Kilgarriff2 May 2007

Overview

Research programme Examples:

Corpus lexicographyWord sketchingCollocationalityThesauruses

The long road


What is language?


What is language? In our heads


What is language? In our heads In texts and sound signals


What is language? In our heads In texts and sound signals Both


Methodology

Study language in our headsCompetenceChomsky“rationalist” (Descartes, Leibniz)


Methodology

Study language in our headsCompetenceChomsky“rationalist” (Descartes, Leibniz)Odd method for objective sciencePractical problems: coverage,

arbitrariness


Methodology

Study text“empiricist” (Locke, Hume)

Physics: forces, matterChemistry: chemicals, bondsLanguage: text, speech signals


It goes against the grain

What is important about a sentence?its meaning

Corpus methodology:Throw away individual sentence

meaningFind patterns


Computer power

Corpora bigger and bigger data sets

Language technology toolslemmatizers, POS-taggers, parsersMachine learning, pattern-finding

18 years of rapid ascent


A virtuous circle

Pattern finding

Linguisticprocessing

Corpus

Lexicon•Part-of-speech tagging

•Parsing

•Lemmatizing

More data →

gets richer eachtime round


Example: corpus lexicography

- four ages


Age 1:Pre-computer

Oxford English Dictionary:• 20 million index cards


Age 2: KWIC Concordances

From 1980 Computerised


Age 2: KWIC Concordance1 arity, which will be used to take a party of under-privileged children to D 2 from outside. You are invited to a party and after a couple of drinks you d 3 tion, we believe politicians of all parties will listen to our views. &equo 4 ould be reaching agreement with all parties concerned, as to which events, 5 lack people. I have certainly been party to one or two discussions amongst 6 . These should be discussed by both parties before entering into the relatio 7 presents They had hosted a cocktail party at Kensington palace, for example 8 akes. By midnight the end-of-course party is in full swing, but most cadet 9 e should be a right for the injured party to terminate the contract. A mana 10 by the Safran Peoples ' Liberation Party. This presents the powerful neigh 11 s. Ahead I could see the rest of my party plodding towards the final slope t 12 cial ethic. The two main political parties - the Tories and the Liberals - 13 ritish successes in Perth The small party of British players competing in th 14 to help control. One member of the party went to summon the rescue team and 15 rket society fashion magazine. The party was held at his flat which was a l 16 security and secrecy than any Tory Party Conference : it seems that bootleg


Age 2: KWIC Concordances

From 1980 Computerised COBUILD project was innovator the coloured-pens method


1 political association 4 person in an agreement/dispute 2 social event 5 to be party to something...3 group of people

1 arity, which will be used to take a party of under-privileged children to D2 from outside. You are invited to a party and after a couple of drinks you d3 tion, we believe politicians of all parties will listen to our views. &equo4 ould be reaching agreement with all parties concerned, as to which events,5 lack people. I have certainly been party to one or two discussions amongst6 . These should be discussed by both parties before entering into the relatio7 presents They had hosted a cocktail party at Kensington palace, for example8 akes. By midnight the end-of-course party is in full swing, but most cadet9 e should be a right for the injured party to terminate the contract. A mana10 by the Safran Peoples ' Liberation Party. This presents the powerful neigh11 s. Ahead I could see the rest of my party plodding towards the final slope t12 cial ethic. The two main political parties - the Tories and the Liberals -13 ritish successes in Perth The small party of British players competing in th14 to help control. One member of the party went to summon the rescue team and15 rket society fashion magazine. The party was held at his flat which was a l16 security and secrecy than any Tory Party Conference : it seems that bootleg

The coloured pens method


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: read all • 500 lines: could read all, takes a long

time• 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listing For collocates of save (>5 hits), window 1-5 words to right of nodeword

word word

forests life

$1.2 dollars

lives costs

enormous thousands

annually face

jobs estimated

money your


Age 4: The word sketch

A corpus-derived one-page summary of a word’s grammatical and collocational behaviour


Age 4: The word sketch

Large well-balanced corpus Parse to find

subjects, objects, heads, modifiers etc

One list for each grammatical relation

Statistics to sort each list, as before


Macmillan English DictionaryFor Advanced Learners

Ed: Rundell, 2002


Euralex 2002


Euralex 2002

Can I have them for my language please


The Sketch Engine Input:

any corpus, any language• Lemmatised, part-of-speech tagged

specification of grammatical relations

Word sketches integrated with corpus query system

Developer: Pavel Rychly, Brno


Users: Dictionary publishers

• Oxford UP, Collins, Chambers, Macmillan

Universities • Teaching, research• Framenet

Language teaching http://www.sketchengine.co.uk/

• Self-registration for free trial account


Collocationality

Which words are most ‘collocational’ Dictionary publishers

Where to put ‘collocation boxes’ Language learners


Verb Freq MLE Prob x log = entropy

Take 2084 -.469

Gain 131 -.169

Offer 117 -.157

See 110 -.150

Enjoy 67 -.104

… … …

Clarify 1 -0.0031

… … …

Total 3730 -3.909

Calculation of entropy for advantage (object relation).



place (17881), attention (8476), door (8426), care (4884), step (4277), advantage (3730), rise (3334), attempt (2825), impression (2596), notice (2462), chapter (2318), mistake (2205), breath (2140), hold (1949), birth (1016), living (953), indication (812), tribute (720), debut (714), button (661), eyebrow (649), anniversary (637), mention (615), glimpse (531), suicide (486), toll (472), refuge (470), spokesman (453), sigh (436), birthday (429), wicket (412), appendix (410), pardon (399), precaution (396), temptation (374), goodbye (372), fuss (366), resemblance (350), goodness (288), precedence (285), havoc (270), tennis (266), comeback (260), farewell (228), prominence (228), go-ahead (202), sip (198),


Thesaurus

a resource that groups words according to similarity


Manual and automatic

Manual Roget, WordNets, many publishers

Automatic Sparck Jones (1960s), Lin (1998) aka distributional two words are similar if they occur in same

contexts Are they comparable?


Thesauruses in NLP sparse data


Thesauruses in NLP sparse data

does x go with y? don’t know, they have never been seen

together New question:

does x+friends go with y+friends indirect evidence for x and y thesaurus tells us who friends are “backing off”


Relevant in: Parsing

PP-attachment conjunction scope

Bridging anaphors Text cohesion Word sense disambiguation (WSD) Speech understanding Spelling correction


Conjunction scope Compare

old boots and shoesold boots and apples




Are the shoes old?




Are the shoes old? Are the apples old?




Are the shoes old? Are the apples old? Hypothesis:

wide scope only when words are similar


(demo)


Words and word senses automatic thesauruses

words



words manual thesauruses

simple hierarchy is appealinghomonyms



words manual thesauruses

simple hierarchy is appealinghomonyms“aha! objects must be word senses”


Problems

Theoretical Practical


Theoretical




Wittgenstein

Don’t ask for the meaning, ask for the use


Practical


Problems Practical

a thesaurus is a toolif the tool organises words senses

you must do WSD before you can use it

WSD: state of the art, optimal conditions: 80%


Problems

“To use this tool, first replace one fifth of your input with junk”


Avoid word senses


Avoid word senses This word has three meanings/senses


Avoid word senses This word has three meanings/senses This word has three kinds of use

well foundedempiricalwe can build on it


Avoid word senses This word has three meanings/senses This word has three kinds of use

well foundedempiricalwe can build on it


sorry, roget


sorry, AI


sorry, AI AI model for NLP:

NLP turns text into meanings AI reasons over meanings word meanings are concepts in an

ontology a Roget-like thesaurus is (to a good

approximation) an ontology Guarino: “cleansing” WordNet

If a thesaurus groups words in their various uses (not meanings) not the sort of thing AI can reason over


“The Future of Search”Berkeley, 4 May 2007

NLP panelPell, Hodjat, RussellThe AI dream for 50 years

• Interpret• Search on meanings

Implicit: word senses


Quine

“No entity without identity”

Foundations for sound science Identity conditions for

word senses? Noword uses? Yes


“linguistics expressions prompt for meanings rather than express meanings”Fauconnier and Turner 2003

It would be nice if … But …


A virtuous circle

Pattern finding

Linguisticprocessing

Corpus

Lexicon•Part-of-speech tagging

•Parsing

•Lemmatizing

More data →

gets richer eachtime round

Word sketching

Thesaurus


The long journey from text towards meaning

Raw text

Pure meaning

Rationalists

Empiricists


The long journey from text towards meaning

Raw text

Pure meaning

Rationalists

Empiricistslemmatizer

POS-taggerparser

thesaurusthematic relations/frame elements


Thank you

http://www.sketchengine.co.uk

Documents

1 The Long Road from Text to Meaning Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex