A Topic map-based ontology IR system versus Clustering-based IR System: A Comparative Study in...

A Topic map-based ontology IR system versus Clustering-based IR System: A Comparative Study in Security Domain

Myongho Yi

Texas Woman’s University, TX, USA, myi@twu.edu

Sam Gyun Oh

SungKyunKwan University, Seoul, Korea, samoh@skku.edu

Agenda

1. Related Works1. Related Works

2. Research Questions2. Research Questions

3. Research Designs3. Research Designs

4. Research Results 4. Research Results

5. Conclusion

Background

• Many information organization approaches such as taxonomy, thesaurus, classification, and ontology have been attempted to provide effective searching.

• Among them, clustering and ontology approaches have received much attention. However, there have not been many studies which compare in terms of user performance.

Three Information Org. Approaches

Term Lists:

Synonym Rings*Authority Files*Glossaries/DictionariesGazetteers*

Natural language Controlled language

Wea

kly- s

truct

u red

Str o

ngly-

stru

ctur

ed

Classification &Categorization: Subject Headings

Classification schemes* Taxonomies*Categorization schemes

Relationship Groups: Ontologies* Semantic networks* Concept maps*Thesauri*

Pick lists*

(Zeng, 2005)

Clustering 2.0

• Classification of data into different subtopic categories. • Clustering shows related items according to their

similarity. • Classify related search results into topic folders• Clustering 2.0

– Remix clustering • Shows other subtle topics

Works in Medical Domain

• Less Polysemes

• Mainly Hierarchical Relationships

• Cancer– Breast Cancer– Prostate Cancer– Colon Cancer– Lung Cancer– Skin Cancer– ….

Norwegian Electronic Health LibraryUnited States Nat’l Lib of Medicine

How about Other Domains?

• Social Sciences

• Humanities

• Polysemes

• Bank– Financial institution– Rely on

Clustering - Limitations

• Relevant results ?– Loosely related associative relationships

• Same / Different category

– Examples• Security

– Information Security» Network Security» PGP» Customers (?)» Valuable (?)» Other Topics (?)

No classified? Loosely related terms?Term Lists?

Purpose of Study

• To measure the efficiency on representations of associative relationships

• To compare the user performance of our Topic Maps-based method with the Clustering-based method.

Related Works

• Yi (2008)

– Compared on Ontology based System to thesaurus based system• 40 subjects, 8 queries, 2 types of queries, search time and recall• An ontology system showed a better recall and search time for relationship

based queries

• Oh (2006)

– Compared on Topic Map-Based Korean Folk Music (Pansori) Retrieval System (TMPRS) to Current Pansori Retrieval System (CPRS)

• Twenty LIS Students in Korea, 7 different search tasks and own query• TMPRS showed higher performance for objective and subjective

measurements in general

• E.K.F. Dang, Luk, Ho, Chan, & Lee, (2008)

– Clustering algorithms

– Partitioning and hierarchical

Research Questions

• Are there recall/precision differences between TMIR and CIR?

• Are there search time differences between TMIR and CIR?

• Are there search steps differences between TMIR and CIR?

Research Design

• Subjects– Information Technology Major Students

• Data Collection– Questionnaire– Screen Recording

• TMIR and CIR– Topic Maps-based Security Information Retrieval

(TMIR) system and Clustering-based Security Information Retrieval (CIR) system.

Research Variables

Two Retrieval Systems

Topic Map-Based Ontology

Information Retrieval System

Clustering-Based Information Retrieval System

Independent Variables

Quantitative Measurement

Search steps, Search Time

Dependent Variables

Search Task Types

Task #

Degree of Relationships

Task

1 Simple Task List all the security software 2 Complex Task Name all the Security engineers who work for Cisco

3 Complex Task Find Vendors providing security training service4 Association and

Cross Reference Related Task

List all the security hardware supported by IBM Consultants

5 Association andCross Reference Related Task

List all software using RSA cryptography and find engineers who specialize in these software packages.

6 Association andCross Reference Related Task

Find security system engineers who specialize in firewall and their supervisor and sale representatives

7 Association andCross Reference Related Task

Assume that your organization is interested in security training. Who will be the right people to contact? Please provide their e-mail addresses

Ontology Development Process

Code using XML Topic Maps (XTM)

Identify Equivalent Relationships

Identify Hierarchical Relationships

Identify Associative Relationships

- Same categories

- Different categories

List Terms by Ontology Engineer

Classify/Categorize by

Ontology Engineer

Add Semantic Relationships by Ontology

Engineer

Normalize by Domain Expert

Implemented by Programmer

Domain Experts

Identify index terms

List the index terms

Do not distinguish between preferred and non-preferred terms

Classify terms

Categorize terms

One term can be in multiple categories

Verify three relationships

Add additional relationships

Ontology Modeling

• Associations– Works for– Maintains– Applied to– Embed in– Provides– Complies with– Designs– Makes– Provides

Embed in

• Cryptography embed in Hardware

Two Retrieval Systems Compared

• Search for “Firewall”• Clustering-based Information Retrieval System

Show Related Terms

Show Related Terms

Two Retrieval Systems Compared

• Clustering-based Information Retrieval System

Firewall Software

Listed

Firewall Software

Listed

No Related Information Provided

Such as Vendor, Engineers for

Firewall

No Related Information Provided

Such as Vendor, Engineers for

Firewall

Two Retrieval Systems Compared

• Converted to the Identical Interface using Omnigator

Two Retrieval Systems Compared

• Search for “Firewall”• Topic Maps-based Information Retrieval System

Show Topic Types and

Associative Relationships

Show Topic Types and

Associative Relationships

Two Retrieval Systems Compared

• Search for “Firewall”• Topic Maps-based Information Retrieval System

Shows the type of information and

related information such as

developers and sales person

Shows the type of information and

related information such as

developers and sales person

Research Results

• There was a significant difference in recall between the two groups.

• The estimate value shows the recall on TMIR was higher than CIR.

• The estimate value also has shown that the search time/search steps in the experimental group was less than in the control group.

Discussion

• There were significant differences between the two groups and in terms of recall, precision, search time, and search steps.

• Overall, recall was higher when performing simple task than when performing complex tasks.

• Performing complex-tasks took more search time than performing simple tasks across the two groups. The control group took more total search time than the experimental group.

Conclusion

• This study illustrates that the positive influences of a Topic map-based ontology IR system are improved recall/precision, shorter search time and search steps for given search tasks than the clustering-based IR system.

• The results of this study attest to the potential of Topic Maps-based ontology to improve information retrieval system performance through better support for associative relationships between terms belonging to different hierarchies by providing explicit relationships among resources.

Q & A

Myongho Yi

Texas Woman’s University, TX, USA, myi@twu.edu

Sam Gyun Oh

SungKyunKwan University, Seoul, Korea, samoh@skku.edu