1 What computers can and cannot do for lexicography or Us precision, them recall Adam Kilgarriff Lexicography Masterclass Ltd and University of Brighton,

1

What computers can and cannot do for lexicography

or

Us precision, them recall

Adam Kilgarriff

Lexicography Masterclass Ltd

and

University of Brighton, UK

27-29 Aug 2003 Adam Kilgarriff: Us precision them recall 2

Outline Precision and recall History of corpus lexicography Natural Language Processing Cyborgs


Find me all the fat cats

a request for information


High recall

Lots of responses Maybe not all good


High precision

Fewer hits Higher confidence



Recall Precision

Computers good bad

People bad good



True in many areas– web searching, google– finding an image to illustrate a talk

Nowhere more so than

lexicography


Lexicography: finding facts about words

collocations grammatical patterns idioms synonyms antonyms meanings translations




Four ages of corpus lexicography


Age 1:Precomputer

Oxford English Dictionary:• 5 million index cards


Age 2: KWIC Concordances From 1980 Computerised COBUILD project was innovator asian-kwic.html the coloured-pens method


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: :read all • 500 lines: could read all, takes a long time,

slow • 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listingFor right collocates of save (>5 hits)

word fr(x+y) fr(y) word fr(x+y) fr(y)

forests 6 170 life 36 4875

$1.2 6 180 dollars 8 1668

lives 37 1697 costs 7 1719

enormous 6 301 thousands 6 1481

annually 7 447 face 9 2590

jobs 20 2001 estimated 6 2387

money 64 6776 your 7 3141


Collocation statistics Which words?

– next word – last word – window, +1 to +5; window, -5 to -1

How sorted? most common collocates --but for most

nouns it's the most salient collocates --how to

measure salience?


Mutual Information

Church and Hanks 1989 How much more often does a word pair

occur, than one might expect by chance “Chance” of x and y occurring together:

p(x) * p(y) Probabilities approximated by

frequencies

p(x) =(approx) f(x)/N


Mutual Information

X fr eat fr X fr eat X

MI* rank

it 1000 400,000

400 1/

1M

3

meat 1000 6000 120 20/

1M

2

sushi 1000 100 8 80/

1M

1

* numbers are log-proportional to MI


Problem mathematical salience = lexicographic

salience? no! higher-frequency items are

lexicographically more salient Solution multiply MI by raw frequency


Mutual Information

X fr eat fr X fr eat X

MI rank

it 1000 400,000

400 1/M 3

meat 1000 6000 120 20/M 2

sushi 1000 100 8 80/M 1

MI x fr

new rank

400/M

3

2400/M

1

640/M

2


Collocation listingFor right collocates of save (>5 hits)

word fr(x+y) fr(y) word fr(x+y) fr(y)

forests 6 170 life 36 4875

$1.2 6 180 dollars 8 1668

lives 37 1697 costs 7 1719

enormous 6 301 thousands 6 1481

annually 7 447 face 9 2590

jobs 20 2001 estimated 6 2387

money 64 6776 your 7 3141


Age-3 collocation statistics: limitations

Lists contain junk unsorted for type --MI lists mix adverbs,

subjects, objects, prepositions

What we really want: noise-free lists one list for each grammatical relation


Age 4: The word sketch Large well-balanced corpus Parse to find

– subjects, objects, heads, modifiers etc

One list for each grammatical relation Statistics to sort each list, as before


Can we do it? high-accuracy parsing is hard lots of NLP work, many parsing frameworks

exist if any parser can handle large corpus, it's

probably good enough

--- sorting, statistics, make us error-tolerant




probably good enough--- sorting, statistics, make us error-tolerant

Poor man’s parsing:– object (of active verb) = last noun in any sequence

of nouns, adjectives, determiners, numbers and adverbs following the verb




probably good enough--- sorting, statistics, make us error-tolerant

Poor man’s parsing:– object (of active verb) = last noun in any sequence

of nouns, adjectives, determiners, numbers and adverbs following the verb


The word sketch British National Corpus (BNC)

– 100 M words, already POS-tagged lemmatized using John Carroll's lemmatizer poor man’s parsing database of 70 million triples

<object, sip, coffee> <subject, arrive, coffee>

<and-or, tea, coffee> <modifier, coffee, instant>

coffee_n.html


Macmillan Dictionary of English for Advanced Leaners, 2002

Editor: Rundell.

Work done 1999. 6000 word sketches

– most common nouns, verbs, adjectives of English HTML files with hyperlinked corpus examples lexicographers used them extensively

– main use of corpus positive feedback


The WASPbench with David Tugwell, UK EPSRC, grant M54971

A lexicographer's workbench runtime creation of word sketches integration with Word Sense Disambiguation

technology output is "disambiguating dictionary" - analysis

of word's meaning into senses, plus computer program for disambiguating contextualised instances of the word

First release now available. http://wasps.itri.brighton.ac.uk/

http://wasps.itri.brighton.ac.uk/






The Sketch Engine Input:

– any corpus, any language Lemmatised, part-of-speech tagged

– specification of grammatical relations Word sketches integrated with Corpus query system

– Supports complex searching, sorting etc First release early 2004




Natural Language Processing

The academic discipline which provides the tools– Also known as Computational Linguistics,

Human Language Technology (HLT), Language Engineering

Good at evaluation of its tools Good news for lexicography:

– identify the best tools, apply them to our corpora


An Anglophone Apology Technology, tools, resources most often

available for English This talk centres on English Other languages often present new

problems– Finding word delimiters for Chinese is hard– Finding bunsetsu for Japanese is hard

Fewer resources available, less work done Recommendation:

– find the local experts for your language


Recap: Lexicography: finding facts about words

collocations grammatical patterns idioms synonyms antonyms meanings translations


Recap: Lexicography: finding facts about words

collocations - sketches grammatical patterns - sketches idioms synonyms antonyms meanings translations


Idioms Extreme case of collocation/multi word

expressions Sequence of workshops on

collocations, MWE Technical terms (of great interest to

technologists, technical): TERMIGHT


Antonyms Essential semantic relation


Antonyms Essential semantic relation

but Justeson and Katz 1995: distributional

evidence for typical antonym pairs– rich men and poor men– the big ones and the small ones– black and white issues

Perhaps antonyms are ‘really’ distributional


Thesauruses Also near-synonyms

– are there any true synonyms? Distributional: which words share same

distributions– if corpus contains

<object, drink, wine>, <object, drink, beer>

– 1 pt similarity between wine and beer– gather all points; find nearest neighbours

Sparck Jones, Lin, Grefenstette


Nearest neighbours In WASPbench Will be generated in Sketch Engine

NOUNS

zebra: giraffe buffalo hippopotamus rhinoceros gazelle antelope cheetah hippo leopard kangaroo crocodile deer rhino herbivore tortoise primate hyena camel scorpion macaque elephant mammoth alligator carnivore squirrel tiger newt chimpanzee monkey


exception: exemption limitation exclusion instance modification restriction recognition extension contrast addition refusal example clause indication definition error restraint reference objection consideration concession distinction variation occurrence anomaly offence jurisdiction implication analogy

pot: bowl pan jar container dish jug mug tin tub tray bag saucepan bottle basket bucket vase plate kettle teapot glass spoon soup box can cake tea packet pipe cup


VERBS

measure

determine assess calculate decrease monitor increase evaluate reduce detect estimate indicate analyse exceed vary test observe define record reflect affect obtain generate predict enhance alter examine quantify relate adjust

boil

simmer heat cook fry bubble cool stir warm steam sizzle bake flavour spill soak roast taste pour dry wash chop melt freeze scald consume burn mix ferment scorch soften


ADJECTIVES

hypnotic

haunting piercing expressionless dreamy monotonous seductive meditative emotive comforting expressive mournful healing indistinct unforgettable unreadable harmonic prophetic steely sensuous soothing malevolent irresistible restful insidious expectant demonic incessant inhuman spooky

pink

purple yellow red blue white pale brown green grey coloured bright scarlet orange cream black crimson thick soft dark striped thin golden faded matching embroidered silver warm mauve damp


Translation Parallel corpora

– Texts and their translations or Comparable corpora

– Matched for source and target (genre and subject matter), not translations

Which L1 words occur in equivalent L1 settings to L2 words in L2 settings?– They are candidate translation pairs

Very hard problem Lots of high quality research




Cyborgs Robots: will they take over? Rod Brooks’s answer:

– Wrong question: greatest advances are in what the human+computer ensemble can do


Cyborgs A creature that is partly human and

partly machine – Macmillan English Dictionary






Cyborgs and the Information Society

The dictionary-making agent is part human (for precision), part computer (for recall).


Treat your computer with respect. You and it can do

great things together.


Lexicographers of the future?

Documents

1 What computers can and cannot do for lexicography or Us precision, them recall Adam Kilgarriff Lexicography Masterclass Ltd and University of Brighton,