Text mining and Indexing: Assessing the Results of

Deeper Indexing for Patent Search

John TaitChief Scientific OfficerIRF

1

Acknowledgements

• Mihai Lupu of the IRF• Jian-Han Zhu of University College

London• Jimmy Huang of York University of

Canada• Giovanna Roda of colleagues in the CLEF

IP team for Matrixware and the IRF• Royal Society for Chemistry for

generously making their Scientific Journal Colelctions available to us

IRF Member Services 2

An apology

The content of this talk was planned on the basis we could discuss the detail of TREC CHEM: over the weekend it became clear NIST policy is that the results should not be made public until the TREC conference in the US in November

• Therefore much detail had to be removed

IRF Member Services 3

Outline

• Introduction to the IRF• TREC CHEM

– 2009– Plans for the future

• Summary and Conclusions

IRF Member Services 4

The IRF

5

The Information Retrieval Facility

A international not-for-profit institution, founded in 2006, based in Vienna, to promote and facilitate research in large scale information retrieval

The IRF Mission

To bridge the gap between the needs of the industry and the academic know-how.To maintain a facility that enables largescale information retrieval and in depthprocessing of data for researchTo bring the latest information retrieval technology to the community of patentprofessionals and other professional searchers.

IRF – Founding Members

The Information Retrieval Facility

A platform initiated by Matrixware which: improves the transfer of knowledge between professionals in Intellectual Property and Information Retrieval andpromotes collaboration between experts on the development of new research methodologies for international patent data .

Distinctive Patent Search Characteristics

High Recall: a single missed document can invalidate a patentSession based: single searchers may involve days of cycles of results review and query reformulationDefendable: Process and results may need to be defended in court

CLEF-IPThe goal of the CLEF-IP track is to investigate

multilingual IR techniques in the Intellectual Property domain.

• Target data >1Mio EPO granted patents documents in three languages: English, German, French

• Tasks prior art search, invalidity search

• Test collection constructed using the available EPO prior art reports

CLEF-IP Track

IRF Member Services

• Scientific Members– Access to data to resources– Project links to industry

• Industrial Members– Consultancy and research in IR and IP

search– Training and support: systems evaluation

semantic computing– Links to academia

IRF Member Services 12

TREC Chemistry Information Retrieval

Track 2009

John TaitChief Scientific OfficerIRF

13

TREC

• Organised by the US Federal Institute of Standards and Technology

• Has run annually since1991– Originally focused on ad hoc text

retrieval with long queries– Regularly extended

• Video• Web• Genomics• Legal

IRF Member Services 14

Origins of TREC CHEM

• IRF approached NIST about using our patent data and computing facilities as a means to promote scientific co-operation

• Jian-Han Zhu then of UK Open University about Chemistry approached NIST about Chemistry

• NIST were interested in domain specific retrieval to follow up Genomic track etc. and helped us get going

IRF Member Services 15

Data

• 1.2 mil. patent files (IRF)

• 59k scientific articles (RSC)

IRF Member Services 16

Tasks

• Technical Survey– Search for all potentially relevant

documents, in both collections.– 18 manually defined and evaluated

topics• Prior Art

– Search for patents that may invalidate a given patent

– 1000 automatically created and evaluated topics (1000 patent files)

17

Participants

• 15 institutions registered to get the data– 6 submitted 31 runs for the TS task:

• University of Applied Science Geneva, Information Retrieval Laboratory of Dalian University of Technology, Fraunhofer SCAI, Milwaukee School of Engineering, Purdue University, York University

– 8 submitted 59 runs for the PA topics:• University of Applied Science Geneva,

Carnegie Mellon University, Information Retrieval Laboratory of Dalian University of Technology,University of Iowa,Fraunhofer SCA, Milwaukee School of Engineering, Purdue University, York University

18

Methods

• Basic vector space model– Different sections, weights on each

section– bm25

• Additional filtering/weighting based on IPC codes

• Linguistic processing– Emphasis on Noun Phrases

• Concept based search• Query expansion

– Using Oscar3, MeSH19

Evaluations

• Technology Survey tasks– 8 chemistry grad students– 5 experts– Each topic evaluated by 2 students and 1

expert• Prior Art tasks

– Automatically evaluated based on citations within patents and family members

20

Initial Results

• Manual evaluations have some conflicting results– Not more than other manually evaluated

topics• Using entity recognition and synonyms

proves successful– Some groups manually extended the

queries• “simple methods” seem to also perform

well (e.g. Lucene-based, bm25)– E.g. for Inferred Average Precision they

reach 97% of highest score• Disclaimer: results analysis is still ongoing

21

TREC CHEM 2010 onwards

• Subject to discussion at TREC in November– Increased numbers of patents– Include images– Task extensions/refinements

• Searching for numerical ranges (independent of unit)

• Searching for specific roles of specific chemical components

• The use of Markush structures

IRF Member Services 22

Summary and Conclusions

• The IRF is promoting collaboration between information retrieval and intelelctual property professionals through promoting evaluations and joint technology development projects

• TREC CHEM has provided an objective and independent means as assessing the effectiveness of technologies on two sorts of retrieval tasks

IRF Member Services 23

Thank you for your attentionAny questions ?

www.ir-facility.orgwww.matrixware.comwww.matrixware.net

IRF Newsletter Chemistry Issue:http://www.ir-facility.org/the_irf/newsletter