Embed Size (px)
Citation preview
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Named Entity Work in IMPACTDr. Frank Landsbergen, Institute for Dutch Lexicology
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Structure of this talk Named Entities (NE‘s): definition NE‘s in OCR and end-user retrieval NE-lexicon building
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Structure of this talk Named Entities (NE‘s): definition NE‘s in OCR and end-user retrieval NE-lexicon building
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
What are named entities? The Examiner (September 1845)
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
What are named entities? The Examiner (September 1845) Named entity: a word or string of
words referring to a proper location, person, or organisation (or date, time, etc.)
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Named Entities (NE‘s): definition NE‘s in OCR and end-user retrieval NE-lexicon building
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
From image to OCR’ed text
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Napoleon
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Napoleon
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Frederic?k? Barbarosa
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Frederic?k? Barbarosa
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Frederic?k? Barbarosa
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Leipzig
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval
On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of thepalace of the Emperor Frederick Barbarossa, and Hanau, atolerably large place containing about 14,000 inhabitants,and the scene of more than one warlike exploit. It wasgallantly defended for nine months during the thirty years'war, and in later days Napoleon, while in hot retreat fromLeipsic, dashed through the allied troops of the AustriansBavarians under its walls. The Queen reachedI i'ankfort at three o'clock in the afternoon, and descendedat the Hotel d'Angleterre, where a body of the City guardwas drawn up to receive her Majesty. In the evening,after dinner, there was a reception or levee, at which theLarl and Countess of Westmoreland, the Burgomaster,and some others of the authorities of Frankfort were pre-sented. Prince Metternich and the King of Bavaria dinedwith the Royal party.
User searches for: Frankfurt
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Retrieval Enable user to find NE’s regardless of historical spelling variation
Leipsic, Dares Salem, Frankfort Improve quality of OCR for NE’s
I i'ankfort, Larl of Westmoreland
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Named Entities (NE‘s): definition NE‘s in OCR and end-user retrieval NE-lexicon building
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Building an NE-lexicon Database of named entities Related entries (wordforms) in the database are linked Entries are enriched (e.g. PND)
AmsterdamAmsterdamsAmsteldamAemsteldamAmstelredamme
Amsterdam
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Building an NE-lexicon Database of named entities Related entries (wordforms) in the database are linked Entries are enriched (e.g. PND)
Frederik Hendrik PND-id: 118590200
FrederikFrederickFredericFredricFrédéricFrederiks
AmsterdamAmsterdamsAmsteldamAemsteldamAmstelredamme
Amsterdam
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Building an NE-lexicon: a 4 step-process1. data collection (NE-lists, text)2. NE-tagging3. enrichment (POS-tagging, lemmatizing, adding name structure, variants)4. database creation
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Building an NE-lexicon: NE-tagging NE-extraction software Stanford University, with added module Statistical NE-recognition:
– Software ‘trains itself’ with manually tagged data– automatic search for features + weights that ‘predict’ NE’s in tagged text– uses trained model to mark NE’s in untagged text
Manual tagging of training data: Attestation Tool
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
NE-recognition Statistical NE-recognition software works very well with modern data. But how
about historical material? Benchmark tests, comparing results for modern & historical data
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
NE-recognition NE-recognition on historical data not necessarily poorer than on modern data
More important is the resemblance between training set and extraction set (time period, genre)
Added spelling variation module improves performance by ‘smoothening’ variation
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
Building an NE-lexicon: enrichment Lemmatization POS-tagging Person name structure Variant matching Link to autority files
NE-variant matching module that links:Historical wordform to modern lemma
Aemsteldam > AmsterdamHistorical wordforms to each other
Frederik, Fredrik, Fréderick
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
The NE-lexicon
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
IMPACT-work on NE‘s so far Toolkit for NE-lexicon building available (NERT, Attestation Tool) NE-lexica of Dutch, English, German
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
IMPACT-work on NE‘s so far Toolkit for NE-lexicon building available (NERT, Attestation Tool) NE-lexica of Dutch, English, German
Hendricus Hamer Hendricus Hamerus
Hendrikkus
Hindrikus
Hendericus
Henderikus
Heudricus
Henricus
Hendrikus
Henriçus
henriecus
Hamer
Amerus
Haamer
Hammer
IMPACT is supported by the European Community under the FP7 ICT Work Programme. The project is coordinated by the National Library of the Netherlands.
ContactFrank Landsbergen
Institute for Dutch Lexicology
Leiden, Netherlands
