1 Corpora, Language Technology and Maltese Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd University of Sussex

1

Corpora, Language Technology and Maltese

Adam KilgarriffLexical Computing LtdLexicography MasterClass LtdUniversity of Sussex

Kilgarriff, Lexical ComputingSlide: 2 Malta, Nov 2006

How do you find out about a language?

Native speakers Dictionaries and Grammars Corpus


Four ages of corpus research


Age 1:Pre-computer

Oxford English Dictionary:• 20 million index cards


Age 2: KWIC Concordances

From 1980 Computerised


Age 2: KWIC Concordance1 arity, which will be used to take a party of under-privileged children to D 2 from outside. You are invited to a party and after a couple of drinks you d 3 tion, we believe politicians of all parties will listen to our views. &equo 4 ould be reaching agreement with all parties concerned, as to which events, 5 lack people. I have certainly been party to one or two discussions amongst 6 . These should be discussed by both parties before entering into the relatio 7 presents They had hosted a cocktail party at Kensington palace, for example 8 akes. By midnight the end-of-course party is in full swing, but most cadet 9 e should be a right for the injured party to terminate the contract. A mana 10 by the Safran Peoples ' Liberation Party. This presents the powerful neigh 11 s. Ahead I could see the rest of my party plodding towards the final slope t 12 cial ethic. The two main political parties - the Tories and the Liberals - 13 ritish successes in Perth The small party of British players competing in th 14 to help control. One member of the party went to summon the rescue team and 15 rket society fashion magazine. The party was held at his flat which was a l 16 security and secrecy than any Tory Party Conference : it seems that bootleg


Age 2: KWIC Concordances

From 1980 Computerised COBUILD project was innovator the coloured-pens method


1 political association 4 person in an agreement/dispute 2 social event 5 to be party to something...3 group of people

1 arity, which will be used to take a party of under-privileged children to D2 from outside. You are invited to a party and after a couple of drinks you d3 tion, we believe politicians of all parties will listen to our views. &equo4 ould be reaching agreement with all parties concerned, as to which events,5 lack people. I have certainly been party to one or two discussions amongst6 . These should be discussed by both parties before entering into the relatio7 presents They had hosted a cocktail party at Kensington palace, for example8 akes. By midnight the end-of-course party is in full swing, but most cadet9 e should be a right for the injured party to terminate the contract. A mana10 by the Safran Peoples ' Liberation Party. This presents the powerful neigh11 s. Ahead I could see the rest of my party plodding towards the final slope t12 cial ethic. The two main political parties - the Tories and the Liberals -13 ritish successes in Perth The small party of British players competing in th14 to help control. One member of the party went to summon the rescue team and15 rket society fashion magazine. The party was held at his flat which was a l16 security and secrecy than any Tory Party Conference : it seems that bootleg

The coloured pens method


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: read all • 500 lines: could read all, takes a long

time• 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listing For right collocates of save (>5 hits)

word freq word freq

forests 6 life 36

$1.2 6 dollars 8

lives 37 costs 7

enormous 6 thousands 6

annually 7 face 9

jobs 20 estimated 6

money 64 your 7


Age 4: The word sketch

A corpus-derived one-page summary of a word’s grammatical and collocational behaviour


Age 4: The word sketch

Large well-balanced corpus Parse to find

subjects, objects, heads, modifiers etc

One list for each grammatical relation

Statistics to sort each list


Macmillan English DictionaryFor Advanced Learners

Ed: Rundell, 2002


Developer: Pavel Rychly, Brno Users:

OUP, Chambers, CUPUniversities for teaching and

research ELT textbook authors

Demo:http://www.sketchengine.co.uk/

• Self-registration for free account


How to develop language technologies?

Introspection Copy others Corpus


Last two decades

Corpora have moved centre-stage forSpellcheckersGrammar checkersAutomatic translationQuestion-answering…

Machine learning from big data


The corpus

EnglishBritish National Corpus100M words, range of text typesLed the worldLate 1980s/ early 1990s

Maltese …


Corpus design: ideals

Very largeMost words, phrases are rare

Cover all types of text All texts labelled by text type, author Available for anyone to use Only real language


Corpus development: pragmatics

What you can get Some sources easy, others hard Often unlabelled

expensive to label Copyright Lots of junk

expensive to clean


Sources

TraditionalBook publishersNewspapers, magazinesOfficial

Web


Traditional

Write to them and askWooing neededAppeal to national prideNewspapers:

• Large

Other sources:• Often small amounts


Web

Lots of data (even for Maltese) Free access

(but copyright) Some formal

Laws, government pages some informal

Chatroom, email


Laws vs web crawl


Agenda

Data cleaning More kinds of text Spelling: standardisation Morphology

inventing invent (v, -ing) Grammar


In sum

Dictionaries, language technology need corpus

MalteseSome components availableMore work needed

Solid foundation for LT is within reach

Documents

1 Corpora, Language Technology and Maltese Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd University of Sussex