1 Evaluating word sketches Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex

1

Evaluating word sketches

Adam KilgarriffLexical Computing LtdLexicography MasterClass LtdUniversities of Leeds and Sussex

Adam Kilgarriff 2 Geneva, April 2010

Overview Research programme Examples:

Case study• Word sketching



What is language?


What is language? In our heads


What is language? In our heads In texts and sound signals


What is language? In our heads In texts and sound signals Both


Methodology

Study language in our headsCompetenceChomsky“rationalist” (Descartes, Leibniz)


Methodology

Study language in our headsCompetenceChomsky“rationalist” (Descartes, Leibniz)Odd method for objective sciencePractical problems: coverage,

arbitrariness


Methodology

Study text“empiricist” (Locke, Hume)

Physics: forces, matterChemistry: chemicals, bondsLanguage: text, speech signals


It goes against the grain

What is important about a sentence?its meaning

Corpus methodology:Throw away individual sentence

meaningFind patterns


Twenty years of rapid ascent

Computer power Corpora

bigger and bigger data sets Language technology tools

lemmatizers, POS-taggers, parsersMachine learning, pattern-finding


A virtuous circle

Pattern finding

Linguisticprocessing

Corpus

Lexicon•Part-of-speech tagging

•Parsing

•Lemmatizing

More data →

gets richer eachtime round


Case study: corpus lexicography

- four ages


Age 1:Pre-computer

Oxford English Dictionary:• 20 million index cards


Age 2: KWIC Concordances

From 1980 Computerised


Age 2: KWIC Concordance

1 arity, which will be used to take a party of under-privileged children 2 from outside. You are invited to a party and after a couple of drinks 3 tion, we believe politicians of all parties will listen to our views.4 ould be reaching agreement with all parties concerned, as to which 5 lack people. I have certainly been party to one or two discussions 6 . These should be discussed by both parties before entering into the7 presents They had hosted a cocktail party at Kensington palace, for8 akes. By midnight the end-of-course party is in full swing, but most 9 e should be a right for the injured party to terminate the contract. A10 by the Safran Peoples ' Liberation Party. This presents the powerful11 s. Ahead I could see the rest of my party plodding towards the final 12 cial ethic. The two main political parties - the Tories and the 13 ritish successes in Perth The small party of British players competing 14 to help control. One member of the party went to summon the rescue15 rket society fashion magazine. The party was held at his flat which16 security and secrecy than any Tory Party Conference : it seems that


Age 2: KWIC Concordances

From 1980 Computerised COBUILD project was innovator the coloured-pens method


1 arity, which will be used to take a party of under-privileged children to D2 from outside. You are invited to a party and after a couple of drinks you d3 tion, we believe politicians of all parties will listen to our views. &equo4 ould be reaching agreement with all parties concerned , as to which events,5 lack people. I have certainly been party to one or two discussions amongst 6 . These should be discussed by both parties before entering into the relatio7 presents They had hosted a cocktail party at Kensington palace, for example8 akes. By midnight the end-of-course party is in full swing, but most cadet9 e should be a right for the injured party to terminate the contract. A mana10 by the Safran Peoples ' Liberation Party . This presents the powerful neigh11 s. Ahead I could see the rest of my party plodding towards the final slope t12 cial ethic. The two main political parties - the Tories and the Liberals -13 ritish successes in Perth The small party of British players competing in th14 to help control. One member of the party went to summon the rescue team and15 rket society fashion magazine. The party was held at his flat which was a l16 security and secrecy than any Tory Party Conference : it seems that bootleg

The coloured pens method


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: read all • 500 lines: could read all, takes a long

time• 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listing For collocates of save (>5 hits), window 1-5 words to right of nodeword

wordword

yourmoney

estimatedjobs

faceannually

thousandsenormous

costslives

dollars$1.2

lifeforests


Age 4: The word sketch

A corpus-derived one-page summary of a word’s grammatical and collocational behaviour


Age 4: The word sketch

Large corpus Parse to find

subjects, objects, heads, modifiers etc

One list for each grammatical relation

Statistics to sort each list, as before


Macmillan English DictionaryFor Advanced Learners

Ed: Rundell, 2002


Euralex 2002


Euralex 2002

Can I have them for my language please


The Sketch Engine

Input: any corpus, any language

• Lemmatised, part-of-speech tagged specification of grammatical

relations Word sketches integrated with

corpus query system

Developer: Pavel Rychly, Brno


Users: Dictionary publishers

• Oxford UP, Collins, Chambers, Macmillan, Cambridge UP

Universities • Teaching, research• Framenet

Language teaching http://www.sketchengine.co.uk/

• Self-registration for free trial account


Lexical Computing Ltd

Since 2003Directors

• Adam Kilgarriff (UK), Pavel Rychly (Cz), • Diana McCarthy (UK, since Oct 2009)

Main activities• Sketch engine service• Corpus development

Research-led


(demo)



10 years 1999-2009

Feedback Good but anecdotal

Formal evaluation


Goal

Collocations dictionary Model: Oxford Collocations Dictionary Publication-quality

Ask a lexicographer For 42 headwords

• For 20 best collocates per headwords “should we include this collocation in a

published dictionary?”


Sample of headwords Nouns verbs adjectives, random High (Top 3000) N space solution opinion mass corporation leader V serve incorporate mix desire Adj high detailed open academic Mid (3000- 9999) N cattle repayment fundraising elder biologist sanitation V grieve classify ascertain implant Adj adjacent eldest prolific ill Low (10,000- 30,000) N predicament adulterer bake bombshell candy shellfish V slap outgrow plow traipse Adj neoclassical votive adulterous expandable


Precision and recall We test precision Recall is harder

How do we find all the collocations that the system should have found?

Current work• 200 collocates per headword

• Selected from

• All the corpora we have

• Various parameter settings

• Plus just-in-time evaluation for 'new' collocates


Four languages, three families

Dutch ANW, 102m-word lexicographic corpus

English UKWaC, 1.5b web corpus

Japanese JpWaC, 400m web corpus

Slovene FidaPlus, 620m lexicographic corpus


User evaluation

Evaluate whole system Will it help with my task

• Eg preparing a collocations dictionary

Contrast: developer evaluation Can I make the system better?

• Evaluate each module separately

• Current work


Components

Corpus NLP tools

Segmenter, lemmatiser, POS-tagger

Sketch grammar Statistics


Practicalities

Interface Good, Good-but

• Merge to good Maybe, Maybe-specialised, Bad

• Merge to bad

For each language Two/three linguists/lexicographers If they disagree

• Don't use for computing performance


Results

Dutch 66% English 71% Japanese 87% Slovene 71%


Corpus evaluation Collocation-finding

Typical corpus task Recall Hold all else constant

Statistic, NLP tools, grammarBest results: best corpus

• (for collocation-finding)

Pomikalek: de-duplication


Other topics Dante

a new lexical database for English

Corpus building (mostly from the web) Instant corpora with WebBootCaT Bigger and better (English)

• BiWeC and New Model Corpus

Corpus Factory (many languages) Corpus comparison, similarity, evaluation Statistics: collocations, keyword lists

Word frequency lists Word senses and lexicography

SADD: semi-automatic dictionary drafting


Thank you

http://www.sketchengine.co.uk


Words and word senses automatic thesauruses

words



words manual thesauruses

simple hierarchy is appealinghomonyms



words manual thesauruses

simple hierarchy is appealinghomonyms“aha! objects must be word senses”


Problems

Theoretical Practical


Theoretical




Wittgenstein

Don’t ask for the meaning, ask for the use


Practical


Problems Practical

a thesaurus is a toolif the tool organises words senses

you must do WSD before you can use it

WSD: state of the art, optimal conditions: 80%


Problems

“To use this tool, first replace one fifth of your input with junk”


Avoid word senses


Avoid word senses This word has three meanings/senses


Avoid word senses This word has three meanings/senses This word has three kinds of use

well foundedempiricalwe can build on it

Documents

1 Evaluating word sketches Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex