Parsalyzer

www.mtcsoft.es

A Linguistic Solution to Perfecting Search Technology

A Linguistic Solution to Perfecting Search Technology

Fernando Moreno-TorresGeneral Manager

MTC Soft

A new tool to improve the filtering options in advanced searching

© MTC Soft

www.mtcsoft.esMadrid – April, 2009 Page 2

1.- MTC Soft

25 years of experience

Our customers:

Software developers

Database design experts

Our products:


The Problem (I): different meanings

European Commission concrete

European Commission - Research: Industrial technologies - Robot ...

Concrete repair The permanence of our built environment can easily be taken for granted. Materials such as concrete, that are used to construct our roads, ...

Concrete tasks or activities contained in White Paper Financial ...

European Commission End 2006. 46) Concrete actions in mortgage credit (follow-up White Paper) European Commission 2007and ...

UNCTAD XI: European Commission calls for concrete action to assist ...

The European Commission will call for concrete global measures to support commodity dependent developing countries and their producers at the 11 ...

European Commission – Information roadshow on the European ...

European citizenship… not just words, concrete rights! - Information roadshow organised by the European Union. European Commission. ...

Results: 967.000


The Problem (II): no relations between words

Tom Cruise bought house

Doctors not fans of Tom Cruise's baby gift - Women's health- msnbc.com

6 Dec 2005 ... Tom Cruise bought a sonogram machine so that he and girlfriend Katie Holmes can see their unborn .... Video: From 'House' to White House ...

He shows them the money - Los Angeles Times

6 May 2007 ... Despite speculation that Tom Cruise and Katie Holmes are having marital problems, he just bought a Beverly Hills home for his family that cost ... The one-story, 13000-square-foot house was built on speculation this year …

Tom Cruise Buys $35 Million Beverly Hills Mansion - Katie Holmes ...

7 May 2007 ... ft. structure in 1937, the house was expanded four years ago and ... have bought a multi-million-dollar house only two minutes away. They plan to move in July. Tom Cruise's $35 million Beverly Hills mansion Photo by:

Tom Cruise Bought Katie Holmes An Engagement Ring Right After ...

Tom Cruise bought wife Katie Holmes an engagement ring after their first date. .... Charlotte Church Kicks Her Boyfriend Out Of House After Heated Argument ...

Results: 169.000


The problem

Too many results, most of then not related to the query

Several-word searches

Common words

¿Why?¿Why?

Search engines don’t discriminate different meanings of wordsSearch engines don’t discriminate different meanings of words

Words are not related in the textsWords are not related in the texts


The solution

Parsing and tagging the textsParsing and tagging the texts

- Discovering the real meaning of every word in every sentence

- Discovering the relations between words

The search engines can use this information: to filter the results to get accurate results

With these “enriched” texts we can give users new possibilities to filter the search using additional options.


The advanced search query

Who? Tom Cruise The sentence subject The actor

What does he do?

To buy The sentence verb The action

Concerning what?

HouseThe sentence direct

objectThe object

Where? In Hollywood (optional) The place

When? Last Month (optional) The time


The new query and the new results

- Tom Cruise has bought a house…

- Tom Cruise and Katie Holmes will buy a new house ….

- Yesterday Tom Cruise bought two dogs and a red house…

This query would find:

- Tom Cruise bought a dog and visited a house in …

- Tom Cruise is very famous. Madonna has bought an expensive house…

- Tom Cruise doesn’t house any friends, because he has bought a car and …

This query would not find:

Tom Cruise is the subject of a sentence in which buy is the verb and house is the direct object


How did we make it? A 10-year project

We perform Word Sense Disambiguation using linguistic knowledge

Based on the linguistic analysis from a philologist and professional translator, working for the European Commission for more than 20 years

It’s the core of our new Automatic Translator (to be released next May 20th)

It’s not Statistical Natural Language Processing It’s not Artificial Intelligence software It’s not the work of software engineers

It’s a design of a philologist, constructed by software developers, MTC Soft, over the last three years


Where it comes from

The translator has 4 main phases or modules.

4PHASE

3PHASE

2PHASE

1PHASE

Symbol identification

Grammatical Analysis

Syntactic Analysis Translation

Parsalyser

VICTOR Translator

EN

GL

ISH

SP

AN

ISH

The first 3, in English, analyze and tag every word of the text. This is the “Parsalyser.”


How does it work?

Germany is supporting the development of infrastructure (especially energy and water supplies), promotion of the economy and employment, advising the PISG on the restructuring of administrative structures in the education and vocational training area and the improvement of the general economic climate.

3PHASE

2PHASE

1PHASE



Syntactic Analysis

Original text+ 140 Processes

+ 500 linguistic rules

The text is analyzed in every process

Every process does its work- Word identification- Capital letter identification- “How” gets special treatment - “Could” gets special treatment

- Disambiguation:-Verb – Noun-Adjective – Pronoun-Verb – Preposition-…

+ 140 Processes


Tagged text- …

In just a few

seconds


Phase 1 – Symbol identification

Tom Cruise has just bought a house that costs $35 million.

3PHASE

2PHASE

1PHASE



Syntactic Analysis

Identification of: words, symbols and expressions Identification of root words (using our dictionary) Assignation of possible grammar functions

Original text:

Words identified:


Phase 2 – Grammatical Analysis3PHASE

2PHASE

1PHASE



Syntactic Analysis

Disambiguation: Performing an exhaustive linguistic analysis

Text after this phase:

Tom Cruise has just bought a house that costs $35 million.Original text:


Phase 3 – Syntactic Analysis3PHASE

2PHASE

1PHASE



Syntactic Analysis

Identification of the syntagms: Both simple and complex Both autonomous and subordinate Identification of syntactic and semantic relations between them

Syntactic functions:

Tom Cruise has just bought a house that costs $35 million.Original text:

Subject Verb Object


An example

On the basis of an annual report from the Commission, it shall also examine the effects of the special arrangements with regard to drugs, including the progress in the fight against drugs made by countries listed in the second annex and, if progress is insufficient, the Commission will consider taking any measures to suspend in whole or in part the application of Article 7, in accordance with the established procedure in Article 32 and after consulting the country concerned.

3PHASE

2PHASE

1PHASE



Syntactic Analysis

Original text:


An example

After Phase 1: Words identified:

3PHASE

2PHASE

1PHASE



Syntactic Analysis


An example

After Phase 2: Words disambiguated:

3PHASE

2PHASE

1PHASE



Syntactic Analysis


An example

After Phase 3: Syntactic functions

3PHASE

2PHASE

1PHASE



Syntactic Analysis


The Dictionary

Página 19

Dictionary Main Statistics:

- 106.634 English words- 31.417 Fixed expressions- 9.587 Open expressions- 20.268 Idiomatic verbs

- Grammar functions: 12.558 Adjectives 4.388 Adverbs 47.150 Substantives 8.558 Verbs (2.590 phrasal verbs)

2.136 acronyms 2.217 incidental phrases

7 professors24 students 4 months

Facultad de Traducción e InterpretaciónGranada University

In collaboration with:


The Dictionary

Every word has its data

10 Dictionaries

Every one has its type of record


State of the “Beta”

3322111

Key words Indexes

Advanced Search Interface

Parser + Tagger = ParsalyserParser + Tagger = Parsalyser

2

3

90%

15%

15%


Indexes

(Simplified example. In the real table, Words are codified using the dictionary)

Key words Indexes 2

Word Grammar Syntactic Sentence File

Function Function

Tom Cruise 1 1 1 123



Tom Cruise 1 3 2 12



Tom Cruise 1 3 12 1.234

Tom Cruise 1 3 33 2.345

buy 1 1 3 1.233

buy 1 1 124 3.432

buy 1 2 44 4.333

buy 3 2 3 234

buy 3 2 23 134

buy 3 2 2 345

buy 3 3 34 4.322

house 1 2 3 234

house 1 2 21 1.456

house 1 3 3 234

house 3 3 12 4.444

1 2 3

Grammar function Substantive Adjective Verb

1 2 3

Syntactic function Subject Verb Direct Object

File: 234

Sentence: 3


User Interface

There is a lot of work done in this area. We has not developed the interface. Just a simple window to test the system. Even so, it is very powerful

3Advanced Search Interface

Only texts with the words relatedX

A first approach:

Tom Cruise bought house

Just a simple “click” to filter most of the results

Only texts with the words related


User Interface

For “advanced” users, a more powerful interface Analyses the sentence Proposes grammar functions Proposes syntactic functions Allows declined forms for every word

3Advanced Search Interface

It works!

Advanced searches:


Plain text

Other uses


SearchEngines

SearchEngines

SemanticWeb

SemanticWeb

DataMining

DataMining

StatisticalAnalysis

StatisticalAnalysis

……


Tagged textPlain text

Other uses



SearchEngines

SearchEngines

SemanticWeb

SemanticWeb

DataMining

DataMining

StatisticalAnalysis

StatisticalAnalysis

……

The parser + tagger can be used for a lot of Internet related tasks


Tagged text

A new Standard?

Why use “plain texts” to start the work with, If you can use “enriched texts”?


Change the “raw material” at the beginning.

These “enriched texts” are: created automatically (no human work) and have an “universal” format (no standard to discuss)

The question is:

www.mtcsoft.es

Thank you for attending

Fernando [email protected]

+34.609.575.000+34.958.215.280

C/ Concepción, 4718009 – Granada (Spain)