Rajan Dhabalia LSP Research Paper

Name: Rajan Dhabalia

LSP : Logic Score Preference

Table of Contain :

(1) Introduction……………………………………………………………………………………………………………………………………………….. 3

(2) Goal of the project……………………………………………………………………………………………………………………………………….4

(3) Project requirement…………………………………………………………………………………………………………………………………….4

User Profile

Tools used

(4) Steps for design quality evaluation………………………………………………………………………………………………………………..6

(5) Identify performance attribute………………………………………………………………………………………………………………………7

(6) Elementary criteria……………………………………………………………………………………………………………………………………….10

(7) Aggregation of preference………………………………………………………………………………………………………………………….…28

(8) Competitive system…………………………………………………………………………………………………………………………………….…34

(9) Result…………………………………………………………………………………………………………………………………………………………....38

(1) EVALUATION REPORT FOR THE SearchEngine PROJECT [SearchEngine.txt] ……………………………………………..38

(2) DETAILED EVALUATION RESULTS FOR THE SearchEngine PROJECT [SearchEngine.lst]……………………………….44

(3) Result of SearchEngines [SearchEngine.res]……………………………………………………………………………………………...70

(4) SUMMARY OF RESULTS FOR THE SearchEngine PROJECT [SearchEngine.sum]…………………………………………..72

1. Introduction The software Quality Analysis is a measure of properties of a piece of software or its specifications. The direct measurement of software quality is quite difficult due to lack of quality factor measurement. To resolve this measurement problem, there is a model which measures the quality of the software in terms of the attributes, specifications and characteristics. This model is known as LSP (Logic Score Preference) .When client gives specifications of the software to the developer then client expects the good quality of software from developers. Hence, to decide the quality of software we can use this LSP model. This model validates following software quality attributes.

(1) Functionality � Suitability � Accuracy � Security � Interoperability � Compliance

(2) Usability � Understandability � Learn ability � Operability

(3) Performance � Processing time � Throughput � Resource consumption

(4) Maintainability (5) Portability (6) Reusability

In LSP, the features are decomposed into above aggregation blocks. And this decomposition continues with in the each block until the all the lowest level features are directly measurable and makes tree of decomposed features. And for each feature, an elementary criterion is defined. And LSP calculates elementary preference for each criterion and then aggregate all of them to calculate final global preference. And this global preference shows the quality of the software. We can calculate global preference for different systems and we can analyze and compare the systems’ quality.

2. Goal of the Project The very first tool used for searching on the internet was Archie in 1990. And from that day the era of search engines begins. And effective search engines such as W2Catalog and Aliweb were introduced in 1993. After that, search engines were improved by adding complexity and functionality. And now days we have efficient and qualitative search engines such as Google, Yahoo Search, bing, Ask, etc. Hence, the goal of this project is to analyze the functionality and the complexity of those search engines. And we will find how these search engines satisfy the user requirements. But before the evaluation, evaluator must know the user requirements. It means evaluator must know that who is the user and what user expects from the system. And next stage is that evaluator must have the expert knowledge in the system. Hence, evaluator can sort out the software attributes very well. And for further stage, evaluator should select the system components which can be compared between two systems. Hence, as being an evaluator we will choose the different search engines to compare their system quality. And we will choose different attributes for the search engine system. And finally we will give different scale of preference to each attribute for every search-engine (Google, Yahoo Search, bing, Ask). And after going through whole process of LSP, we will find out global preference of each search-engine and analyze their quality.

In this project we will put more focus on building a comprehensive evaluation model for five search engines: Google, Yahoo Search, bing, Ask, altavista. This model aggregates all the features those reflect the functionality and usability of the search engines and generates a compound indication for the overall quality. Hence finally it reflects the measurement of user satisfaction for all aspects of the search-engines.

3. Project Requirements Here we discuss initial requirements to evaluate this model.

3.1 User Profile To evaluate Logic score preference model for different search-engine, we need profile of user that uses these search engine. And also we need to have purpose of user to use search-engine. User: Student Purpose:

� Search study related documentations � Research papers � Presentation slides [pdf, ppt, etc. files]

� Search books on web � Search images, videos and audio related to projects of study. � Search geographical location of universities or any other places

3.2 Tools Used

� LSPcalc128

� ANSY

� Google Chrome 4.1

� Internet Explorer 7.0

� Secure Shell Client

4. STEPS FOR DESIGN QUALITY EVALUATION Step 1: Develop a hierarchical model for quality characteristics and attributes (i.e. A1 …. An): Such as so with the help of them, we can compare the systems. This step defines and specifies the quality characteristics and attributes, grouping them into a model. For each quantifiable attribute Ai, model associates a variable Wi, which is a weight of that attribute in the system and it can take a real value. - So, for search engine I have developed attribute-tree [4.1] Step 2: Define criterion function for each attribute, and apply attribute measurement for each search-engine. Elementary evaluation criteria specify how to measure quantifiable attributes. The result is an elementary preference, which can be interpreted as a Degree of satisfied requirement. For each attribute, it is necessary to establish an acceptable range of values and define a function, called the elementary criterion. This function maps the measured value in the numerical domain. - So, for search-engine I have developed elementary preference and attribute measurement values in section [5.1] Step 3: - Evaluating elementary preferences - Logic aggregation of preferences

- So, for search engine I have developed logic aggregation preference using pictorial diagram. Aggregators are chosen based on the user needs which are expressed as relationship between inputs to be simultaneity, replaceability and neutrality. There are some inputs which are to be satisfied simultaneously and finally we aggregate the preferences and compute the overall suitability of the system.

Step 4:

- Evaluating competitive functions

- Here, we have five different such engines such as Google, Yahoo Search, bing, Ask, altavista.

And we will evaluate competitive feature analysis for each above search-engines.

Step 5:

- Ranking and selection of the best system and analysis of the system

- And finally using LSPCalc, we can give rank to each search-engine according to their global

reference. And select best among them.

- How to use LSPCalc128 tool ?

Ans: Create .cri and .dat files

� Create a directory in the projects folder named with your project

� Create two files as projectname.cri and projectname.dat

� By default the LSPcalc128 looks into projects folder to search for all the projects

� And then we can open any client shell to access the LSPcalc128

Run the executable and follow the instructions on the screen of this tool.

5. Identifying performance attributes

5.1: System attribute tree

Attribute tree

1.1 Searching Input

1.1.1 Text Searching

1.1.1.1 Keyword Searching

1.1.1.1.1 Search with one keyword

1.1.1.1.2 Search with group of keywords

1.1.1.2 Statement Searching

1.1.1.2.1 Search exact statement

1.1.1.2.2 Search part of statement

1.1.1.2.3 Case sensitive Search

1.1.1.2.4 Ignore stop words (the,a,an,..etc)

1.1.1.3 Search with different language

1.1.1.4 Numeric Expression Search

1.1.1.4.1 Search numeric expression in documents

1.1.1.4.2 Compute numeric expression

1.1.1.5 Relational Searching

1.1.1.5.1 Abbreviation Search

1.1.1.5.2 Synonyms Search

1.1.1.5.3 Stemming Search

1.1.1.5.4 Misspell Corrected Search

1.1.1.6 Operator Searching

1.1.1.6.1 Including Search (using ‘+’ operator)

1.1.1.6.2 Excluding Search (using ‘-‘ operator)

1.1.1.6.3 Combinational Search (using ‘*’ operator)

1.1.1.6.4 Optional Search (using ‘OR’ operator)

1.1.2 Multimedia Search

1.1.2.1 Image Searching

1.1.2.1.1 Filename search

1.1.2.1.2 Image link search

1.1.2.1.3 Adjacent text search

1.1.2.2 Video Searching

1.1.2.2.1 Popularity based Search

1.1.2.2.2 Content based Search

1.1.2.2.3 Content Controlled Search

1.1.2.3 Audio Searching

1.2 Searching filter

1.2.1 Security filter

1.2.1.1 Pages with Spam, doorway

1.2.1.2 Duplicate Content

1.2.2 Citation filter

1.2.2.1 Adult content filter

1.2.2.2 Casino content filter

1.2.3 Domain filter

1.2.3.1 Pages related to same content

1.2.3.2 Pages related to same website

1.2.3.3 Linked page

1.2.4 Extended filter

1.2.4.1 RSS support pages

1.2.4.2 Usage rights

1.2.4.3 Numeric range filter

1.2.4.4 field to search keyword [title,text,URL,Link]

1.2.5 File specific filter [pdf, word, excel sheet,..]

1.2.6 Broken link filter

1.2.7 Time filter

1.2.7.1 Page created time

1.2.7.2 Recent update time

1.2.8 Location filter

1.2.8.1 Location of searching user [i.e. weather forecast]

1.2.8.2 Location of country

1.3 Specific activity Search

1.3.1 Weather

1.3.2 Blog

1.3.3 Movie time

1.3.4 Sport score

1.3.5 Stock price

1.3.6 Literature [books,..]

1.3.7 Maps

2. Usability

2.1 Interface usability

2.1.1 Interface visibility

2.1.2 Operability

2.1.3 Customization

2.1.3.1 Customization of page size

2.1.3.2 Customization of page rank

2.2 Result Evaluation

2.2.1 Result visibility

2.2.2 Accessibility of results

2.2.3 Availability of cached result

2.3 User guide

2.3.1 Online help

2.3.2 Manual user-guide

2.3.3 FAQ

2.3.4 Related tutorial material

3. Performance

3.1 Loading time

3.1.1 Load page time

3.1.2 Automatic search suggestion time

3.1.3 Result evaluation

3.1.3.1 Time to evaluate best result

3.1.3.2 Time to evaluate top N result

3.2 Resource Consumption

4. Reliability

4.1 User satisfaction

4.1.1 Popular pages results

4.1.2 High rank pages results

4.1.3 Coverage of user need

4.2 Confusion matrix

4.2.1 Accuracy

4.2.2 Precision

4.2.3 Recall

4.2.4 Specificity

6 : Elementary Criteria

Point additive discrete absolute criteria

[6.1] 1.1.1 Text Searching

Keyword Searching Weight

Search with one

keyword

- Search engine must search text with the single keyword. And that

single word may relate to any issue. It may from any study subjects

(Computer, Mathematics, Mechanical, etc..), any dictionary word, or

any un-traditional word.

- So, search engine should have capability to cover N no of words.

- Efficiency for Dictionary words - 5

- Efficiency for Non-Dictionary words - 5

50

Search with group of

keywords

- Search engine should have capability to correlate group of keywords

and find combination of these keywords on the web.

- Find all words of group – 8

- Find some of words of group - 2

50

Statement Searching

Search exact

statement

- Search REL= 100 * S/ S max

Efficiency to search exact expression on web as user wants

40

Search parts of

statement


Efficiency to search all divided parts of the expression on web.

60

0 10 20 30 40 50 60 70 80 90 100

Search with one keyword [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Group of keywords [Rank of Accuracy]

1 5 10

1

0 10 20 30 40 50 60 70 80 90 100

Search exact statement [Level of efficiency]

10 5

0 10 20 30 40 50 60 70 80 90 100

Search part of statement [Level of efficiency]

1 10 5

Case sensitive

Search

- if search-engine is case sensitive - 0

- otherwise - 1

20

Ignore stop words

(the,a,an,..etc)

- Ignore REL= 100 * Ignore/ Ignore max

Efficiency to search by removing remove words for efficient search

80

Search with different

language

- Number of languages which search engine can support.

20

Numeric Expression Search

Search numeric

expression in

documents


Efficiency to search Numeric expression

i.e. population of country

80

Compute numeric

expression

- Number of mathematical expression search engine can compute.

- i.e. Mathematical operation +,-,*,/,sinX,tanX,sqrt,etc..

20

0 10 20 30 40 50 60 70 80 90 100

Case sensitive Search [Yes=1 and No=0]

1 0

0 10 20 30 40 50 60 70 80 90 100

Ignore stop words [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Search with different languages[No of languages]

51 1 25

0 10 20 30 40 50 60 70 80 90 100

Numeric exp in document [Level of efficiency]

1 10 5

Compute numeric Exp [No of Expression]

30 4 15

Relational Searching

Abbreviation Search - AS REL= 100 * AS/ AS max

Efficiency to search Abbreviation on web as user wants

20

Synonyms Search - SS REL= 100 * SS/ SS max

Efficiency to search synonyms of searching word

30

Stemming Search - SS REL= 100 * SS/ SS max

Efficiency to search with stemming technique for efficient search.

20

Misspell Corrected

Search

- MS REL= 100 * MS/ MS max

Efficiency to search with correction of misspell word

30

Operator Searching

Including Search

(using ‘+’ operator)

- IS REL= 100 * IS/ IS max

Efficiency to search multiple expression with the operator ‘+’

0 10 20 30 40 50 60 70 80 90 100

Abbreviation Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Synonyms Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Stemming Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Misspell Corrected Search [Level of efficiency]

1 10 5

55

Excluding Search

(using ‘-‘ operator)

- ES REL= 100 * ES/ ES max

Efficiency to search with subtraction facilities , so user can search

specific part from whole part of expression

10

Combinational

Search (using ‘*’

operator)

- CS REL= 100 * CS/ CS max

Efficiency to search combinational search-expression on web as user

wants

15

Optional Search

(using ‘OR’

operator)

- OS REL= 100 * OS/ OS max

Efficiency to search with “Optional search” facility on web as user

wants

20

Image Searching

Filename search - FNS REL= 100 * FNS/ FNS max

Efficiency to search image as its file name on the web

20

0 10 20 30 40 50 60 70 80 90 100

Including Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Excluding Search [Level of efficiency]

1 10 5

1

0 10 20 30 40 50 60 70 80 90 100

Combination Seach [Level of efficiency]

10 5

0 10 20 30 40 50 60 70 80 90 100

Optional Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Filename Search [Level of efficiency]

1 10 5

Image link search - ILS REL= 100 * ILS/ ILS max

Efficiency to search Image on web as user wants

30

Adjacent text search - ATS REL= 100 * ATS/ ATS max

Efficiency to search image using adjacent text with target image on the

web.

50

Video Searching

Popularity based

Search

- VPS REL= 100 * VPS/ VPS max

Efficiency to search video as its popularity

50

Content based

Search

- VCS REL= 100 * VAS/ VAS max

Efficiency to search video using content related to that video.

50

Content Controlled

Search

- VCCS REL= 100 * VCCS/ VCCS max

Efficiency to search video with some specific user defined criteria.

- Funny, drama, Action, etc..

25

0 10 20 30 40 50 60 70 80 90 100

Image link Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Adjacent text Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Popularity based Search [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Content based Search [Level of efficiency]

1 10 5

Content Controlled Search [Level of efficiency]

1 10 5

Audio Searching - AudioS REL= 100 * AudioS/AudioS max

Efficiency to Search audio as per user requirement.

30

[6.2] 1.2 Searching Filter

Security filter Weight

Pages with Spam,

doorway

- SSS REL= 100 * SSS/ SSS max

Efficiency of Security Spam filter is to prevent search of web-sites

which cause to download spam and doorway in user system.

55

Duplicate Content - DCS REL= 100 * DCS/ DCS max

Efficiency to prevent searching duplicate content in fraud web-

sites to follow copyright right rule.

45

Citation filter

Adult content filter - ACS REL= 100 * ACS/ ACS max

Efficiency of filter to prevent Searching Adult content on the web.

90

0 10 20 30 40 50 60 70 80 90 100

Audio Searching [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Pages with Spam,doorway [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Duplicate Content [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Adult Content filter [Level of efficiency]

1 10 5

Casino content filter - CCS REL= 100 * CCS/ CCS max

Efficiency of filter to prevent Searching Casino and gambling

content on the web.

10

Domain filter

Pages related to same

content

- DPS REL= 100 * DPS/ DPS max

Efficiency of filter to search pages which has same kind of

contents.

45

Pages related to same

website

- DWS REL= 100 * DWS/ DWS max

Efficiency of filter to pages which are related to same website

domain.

35

Linked page - DLS REL= 100 * DLS/ DLS max

Efficiency of filter to search pages which are linked with each-

other.

20

0 10 20 30 40 50 60 70 80 90 100

Casino content filter [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Content Related Pages [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Website related website [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Linked Page [Level of efficiency]

1 10 5

Extended filter

RSS support pages - RSS REL= 100 * RSS/ RSS max

Efficiency of filter to search RSS support pages.

35

Usage rights - US REL= 100 * US/ US max

Efficiency of filter to give access to user who has usage right for

target searched page.

30

Numeric range filter - NS REL= 100 * NS/ NS max

Efficiency of filter to search with sorting facility [For numeric

range sort]

35

field to search keyword

[title,text,URL,Link]

Search with specific field

- URL of page

- Title

- Specific page text

- Link of the page

55

File specific filter [pdf,

word, excel sheet,..]

- Ability to find out different kind of file on the web.

- i.e. .ppt, .doc, .pdf, etc..

35

0 10 20 30 40 50 60 70 80 90 100

RSS support pages [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Usage rights [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Numeric range filter [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Field to search keyword [No of Field]

4 1 2

Broken link filter - BLS REL= 100 * BLS/ BLS max

Efficiency of filter to prevent to search broken link pages or pages

no longer exist.

15

Time filter

Page created time - PTS REL= 100 * PTS/ PTS max

Efficiency of filter to search page with created year or time for

that page.

20

Recent update time - RUS REL= 100 * RUS/ RUS max

Efficiency of filter to search page with recent update result.

80

Location filter

Location of searching user

[i.e. weather forecast]

- LS REL= 100 * LS/ LS max

Efficiency of filter to search pages related to geographic location

of user.

30

0 10 20 30 40 50 60 70 80 90 100

File specific filter [No of types]

30 1 15

0 10 20 30 40 50 60 70 80 90 100

Broken link filter [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Page created time [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Recent update time [Level of efficiency]

1 10 5

Location of user [Level of efficiency]

Location of country - LCS REL= 100 * LCS/ LCS max

Efficiency of filter to search pages related to country to which

user belongs.

70

[6.3] 1.3 Specific activity Search

Weather - WS REL= 100 * WS/ WS max

Efficiency to search efficient weather result.

8

Blog - BS REL= 100 * BS/ BS max

Efficiency to search Blogs on web as user wants

15

Movie time - MS REL= 100 * MS/ MS max

Efficiency to search Movie-timing of cinemas on web as user

wants

8

Sport score - SSS REL= 100 *SSS/ SSS max 20

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Location of Country [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Wether [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Blog [Level of efficiency]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Movie time [Level of efficiency]

1 10 5

Efficiency to search latest different sports score on web as user

wants

Stock price - SPS REL= 100 * SPS/ SPS max

Efficiency to search latest stock-price on web as user wants

14

Literature

[books,..]

Efficiency to search different field related books and literatures on

web as user wants

� author – 3 points

� content – 4 points

� publisher – 3 points

15

Maps Map scoring points

� Public transportation trace root – 2

� Driving direction – 2.5

� Walking root – 0.5

� Save Map – 0.5

� Write Map description – 0.5

� Review of place - 1

� Search place by place-properties - 1

� Printable Map – 1

� Maps for mobile or handy-instruments - 1

20

0 10 20 30 40 50 60 70 80 90 100


1 10 5

0 10 20 30 40 50 60 70 80 90 100


1 10 5

0 10 20 30 40 50 60 70 80 90 100


10 1 5

0 10 20 30 40 50 60 70 80 90 100


10 1 5

[6.4] 2. Usability

Interface usability Weight

Interface visibility - Specification of visibility

� Textbox to search – 3

� Target Search Button – 3

� Select search options [image,video] – 1

� Without redundant component – 1

� Easy and Clear - 2

40

Operability - Specification of visibility

� No of Input component – 4

� No of click to search – 3

� Simple Searching criteria – 3

60

Customization

Customization of page

size

- Facility to customize page according to user need

� No of Results per page – 8

� Searching customization – 1.5

� Page-Theme selection – 0.5

20

Customization of page

rank

- CPR REL= 100 *CPR/ CPR max

Efficiency to give importance of user’s taste[Using users’ visits on

the pages] and give high page rank count those pages and show

those pages on top results.

80

0 10 20 30 40 50 60 70 80 90 100

Interface visibility [Level of visibility]

10 1 5

0 10 20 30 40 50 60 70 80 90 100

Operability [Level of user operability]

10 1 5

0 10 20 30 40 50 60 70 80 90 100

Page size[Level of customization]

10 1 5

Result Evaluation

Result visibility - Result visibility

� Top N result on the page – 3

� Easy to access – 5

� Result without redundant data on page – 2

20

Accessibility of results - Accessibility

� No of click – 4

� Brief info of result on result page – 3

� Result according to ranking – 3

80

Availability of cached

result

- CR REL= 100 *CR/ CR max

Efficiency to store and use cache result for next result evaluation.

20

User guide

Online help - ONH REL= 100 *ONH/ ONH max

Availability of online help to guide users.

20

0 10 20 30 40 50 60 70 80 90 100

Page Rank [Level of page rank algorithm]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Result visibility [No of result per page]

10 1 5

0 10 20 30 40 50 60 70 80 90 100

Accessibility of Results [Level of accessibility]

10 1 5

0 10 20 30 40 50 60 70 80 90 100

cached Result[Level of efficiency]

1 10 5

Online Help [Level of satisfied Material]

1 10 5

Manual user-guide - UG REL= 100 *UG/ UG max

Availability of manual user-guide to guide users.

30

FAQ - Frequent ask question

No of FAQ provided by the search engine to solve user doubts.

20

Related tutorial

material

- TM REL= 100 *TM/ TM max

Availability of Related tutorial materials to guide users.

30

0 10 20 30 40 50 60 70 80 90 100

Manual user-guide [Level of Quality of Guide ]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Frequent Ask Questions [No of Questions]

45 5 25

0 10 20 30 40 50 60 70 80 90 100

Tutorial Material [Rank of Material Quality ]

1 10 5

`

[6.5] 3. Performance

Loading time Weight

Load page time - Time taken by page[Search-engine] to load on user system

[in Second]

70

Automatic search

suggestion time

- Time taken by suggestion-search to assist user for searching

[in Second]

30

Result evaluation

Time to evaluate best

result

- Time to evaluate result for target search.

[in Second]

80

Time to evaluate top N

result

- Time taken to evaluate top N result [in Second]

20

Resource

Consumption

- Memory occupied by the HTML source page of the search-

engine.

[Kilo-bytes]

30

0 10 20 30 40 50 60 70 80 90 100

Load page time [seconds]

4 12 8

0 10 20 30 40 50 60 70 80 90 100

Suggestion Time [Seconds]

0.5 3 1

0 10 20 30 40 50 60 70 80 90 100

Time to having result [Seconds]

4 18 11

0 10 20 30 40 50 60 70 80 90 100

Time to having top N Results [ Seconds]

0.9 16 8

0 10 20 30 40 50 60 70 80 90 100

Resource Consumption [Page Size [kb]]

20 520

[6.6] 3. Reliability

User satisfaction

Popular pages results - Efficiency for reliable search

� Popular pages - 4

� Informative pages - 3

� Trustworthy pages - 1

� Ability to search target page of specific web-site

domain rather Home Page – 2

30

High rank pages

results

- Efficiency to computing page rank and evaluate result using

ranking

20

Coverage of user need - Efficiency to satisfy and cover need of user

50

Confusion matrix for result

Accuracy - Accuracy of the Result

20

Precision - Precision for evaluated result for a target search

30

0 10 20 30 40 50 60 70 80 90 100

Popular Pages Results [Level of reliability]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Rank pages results [Level of reliability]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Coverage need [Level of user satisfaction]

1 10 5

0 10 20 30 40 50 60 70 80 90 100

Accuracy [% accuracy]

1 100 50

0 10 20 30 40 50 60 70 80 90 100

Precision [% precision]

1 100 50

Recall - Recall for evaluated result for a target search

20

Specificity - Specification for evaluated result for a target search

30

0 10 20 30 40 50 60 70 80 90 100

Recall [% recall]

1 100 50

0 10 20 30 40 50 60 70 80 90 100

Specificity [% Specificity]

1 100 50

7. Ranking and Aggregation of the preferences

The system requirement tree has 73 performance variables as listed above. We make a weighted graph of

the performance variables and using aggregators .

7.1 Functionality

7.1.1 Text Searching :

[Fig: 1 Functionality: Text-Searching]

7.1.2 Multimedia Searching

[Fig: 2-Functionality: Multimedia Searching]

7.1.3 Searching Filter

[Fig: 3 – Functionality: Searching Filter]

7.1.4 Specific activity Search

[Fig: 4 – Functionality: Specific activity Search]

7.1.5 Functionality :

[Fig: 5 – Functionality]

7.2.1 Usability

[Fig: 6 Usability]

7.3.1 Performance:

[Fig: 7 Performance]

7.4.1 Reliability

[Fig: 8 Reliability]

7.5 Global preference

[Fig: 9 Global Preference]

8.1 Competitive System :

Attributes

Google

Yahoo! Search

Bing

Ask

alavista

1.1.1.1.1 Search with

one keyword

1

2

2

3

3

1.1.1.1.2 Search with

group of keywords

1

2

2

3

3

1.1.1.2.1 Search

exact statement

1

1

2

2

3

1.1.1.2.2 Search part

of statement

1

2

2

2

3

1.1.1.2.3 Case

sensitive Search

1

1

1

1

1

1.1.1.2.4 Ignore stop

words

(the,a,an,..etc)

1

1

1

1

1

1.1.1.3 Search with

different language

46

41

42

6

2

1.1.1.4.1 Search

numeric expression

in documents

2

2

2

2

2

1.1.1.4.2 Compute

numeric expression

30

4

20

6

4

1.1.1.5.1

Abbreviation Search

1

1

1

2

2

1.1.1.5.2 Synonyms

Search

1

1

1

2

2

1.1.1.5.3 Stemming

Search

1

1

2

2

2

1.1.1.5.4 Misspell

Corrected Search

1

1

1

2

2

1.1.1.6.1 Including

Search (using ‘+’

operator)

1

2

2

1

2

1.1.1.6.2 Excluding

Search (using ‘-‘

operator)

3

3

3

3

3

1.1.1.6.3

Combinational

Search (using ‘*’

operator)

2

3

3

3

3

1.1.1.6.4 Optional

Search (using ‘OR’

1

3

1

2

2

operator)

1.1.2.1.1 Filename

search

2

2

2

3

3

1.1.2.1.2 Image link

search

2

2

2

2

2

1.1.2.1.3 Adjacent

text search

1

1

2

2

2

1.1.2.2.1 Popularity

based Search

1

2

1

2

2

1.1.2.2.2 Content

based Search

1

1

1

1

1

1.1.2.2.3 Content

Controlled Search

2

2

2

2

2

1.1.2.3 Audio

Searching

2

2

2

2

2

1.2.1.1 Pages with

Spam, doorway

1

1

1

2

1

1.2.1.2 Duplicate

Content

2

3

3

3

2

1.2.2.1 Adult content

filter

2

2

2

2

2

1.2.2.2 Casino

content filter

3

4

3

4

4

1.2.3.1 Pages related

to same content

3

3

2

3

3

1.2.3.2 Pages related

to same website

1

1

1

1

1

1.2.3.3 Linked page

2

2

2

2

2

1.2.4.1 RSS support

pages

2

3

2

3

3

1.2.4.2 Usage rights

1

2

1

2

2

1.2.4.3 Numeric

range filter

1

3

3

3

3

1.2.4.4 field to

search keyword

[title,text,URL,Link]

4

3

4

4

3

1.2.5 File specific

filter [pdf, word,

excel sheet,..]

30

26

27

24

23

1.2.6 Broken link

filter

4

3

3

4

3

1.2.7.1 Page created

time

4

3

3

4

3

1.2.7.2 Recent

update time

2

2

2

2

2

1.2.8.1 Location of

searching user [i.e.

weather forecast]

2

3

2

4

4

1.2.8.2 Location of

country

2

2

2

3

4

1.3.1 Weather

1

2

2

3

2

1.3.2 Blog

1

2

2

3

3

1.3.3 Movie time

3

1

2

3

3

1.3.4 Sport score

2

1

1

2

2

1.3.5 Stock price

2

1

2

2

3

1.3.6 Literature

[books,..]

9

9

9

8

8

1.3.7 Maps

10

8

9

7

7

2.1.1 Interface

visibility

10

9

9

9

9

2.1.2 Operability

10

10

10

9

9

2.1.3.1

Customization of

page size

10

10

10

10

10

2.1.3.2

Customization of

page rank

2

2

2

3

3

2.2.1 Result

visibility

10

8

9

8

8

2.2.2 Accessibility of

results

10

9

9

9

9

2.2.3 Availability of

cached result

2

2

2

3

2

2.3.1 Online help

1

1

1

2

1

2.3.2 Manual user-

guide

2

2

2

3

2

2.3.3 FAQ

30

28

30

27

28

2.3.4 Related

tutorial material

1

1

1

2

2

3.1.1 Load page time

2

2

2

2

2

3.1.2 Automatic

search suggestion

time

0.5

0.5

0.5

0.5

0.5

3.1.3.1 Time to

evaluate best result

3.5

3.5

3.5

4

4.5

3.1.3.2 Time to

evaluate top N result

3

3

3

3.5

4.0

3.2 Resource

Consumption

76

142

55

360

50

4.1.1 Popular pages

results

1

1

1

3

6

4.1.2 High rank

pages results

1

2

2

3

7

4.1.3 Coverage of

user need

2

3

3

4

3

4.2.1 Accuracy

90

75

85

70

75

4.2.2 Precision

95

80

85

80

80

4.2.3 Recall

88

78

82

78

78

4.2.4 Specificity

93

85

88

84

84

9.1 Results :

9.1.1 EVALUATION REPORT FOR THE SearchEngine PROJECT [SearchEngine.txt]

This report presents the evaluation results for the following 5 competitive

systems:

1. Google

2. Yahoo

3. bing

4. Ask

5. altavista

The evaluation is based on 71 elementary criteria grouped in the following 4

major groups:

1. Functionality

2. Usability

3. Performance

4. Reliability

This summary includes two parts: (1) System Comparison and Ranking, and

(2) Survey of Individual Systems. Deatailed numerical results can be found in

the report entitled "Detailed Evaluation Results of the SearchEngine Project".

(1) System Comparison and Ranking

---------------------------------

The global preference of a system indicates the global percentage of satisfied

requirements. Therefore, the best system has the highest global preference. The

ranking of competitive systems is based on decreasing global preferences, as

follows:

1. 94.89% Google

2. 89.78% bing

3. 88.18% Yahoo

4. 78.42% altavista

5. 77.35% Ask

Therefore, the best system is Google.

This system satisfies 94.89% of the requirements specified by evaluation criteria.

The absolute value of global preference depends both on the quality of each system

and the level of demand imposed by the evaluation criterion function. So, low global

preferences may sometimes reflect too demanding criteria. The relative ranking of

competitive systems is based on normalized preferences so that the best system has

the normalized global preference of 100%. Following is the ranking according to

normalized preferences:

1. 100.00% Google

2. 94.62% bing

3. 92.93% Yahoo

4. 82.65% altavista

5. 81.51% Ask

The relative differences between systems can be interpreted as follows:

System Google dominates system bing in 66.20% of inputs

System Google dominates system Yahoo in 72.54% of inputs

System Google dominates system altavista in 81.69% of inputs

System Google dominates system Ask in 83.80% of inputs

System bing domantes system Yahoo in 59.86% of inputs

The reasons for a specific value of global preference can be explained

by investigating the quality of all major components of the evaluated

systems. Following is the survey of preferences of 4 major system

components: Functionality, Usability, Performance, and Reliability.

Systems Functionality Usability Performance Reliability

-----------------------------------------------------------------------

Google 93.79 96.95 95.82 93.87

Yahoo 89.21 88.40 90.87 85.05

bing 90.12 88.86 97.22 86.18

Ask 82.26 85.55 63.49 73.33

altavis 81.20 88.10 96.62 60.97

-----------------------------------------------------------------------

(2) Survey of Individual Systems

--------------------------------

This survey highlights the strongest and the waekest components of

all evaluated systems. In particular, the survey includes lists of

the weakest components that are primary candidates for improvements.

This is an analysis of relative performance and for high quality

systems the weakest component can still satisfy a substantial

percentage of user's requiremenmts. Therefore, improvements are not

equally urgent for all systems. They are primariliy needed for systems

having a relatively low global preference.

Google

This system satisfies 94.89% of user's requirements. The best subsystem

of Google is Usability.

The best subsystem satisfies 96.95% of specified requirements.

The weakest subsystem of Google is Functionality.

The weakest subsystem satisfies 93.79% of specified requirements.

Weak components of this system are components that are rated

below the global preference. These are components that primarily

need improvement. Following is the sorted list of weak components,

starting with the weakest component:

ID X E[%] Elementary criterion

-------------------------------------------------------------

233 30.00 62.50 FAQ

1271 4.00 66.67 Page created time

126 4.00 66.67 Broken link filter

11162 3.00 77.78 Excluding Search (using ‘-‘ operator)

1222 3.00 77.78 Casino content filter

133 3.00 77.78 Movie time

1231 3.00 77.78 Pages related to same content

3132 3.00 86.09 Time to evaluate top N result

423 88.00 87.88 Recall

32 76.00 88.80 Resource Consumption

11163 2.00 88.89 Combinational Search (using ‘*’ operator)

1233 2.00 88.89 Linked page

1241 2.00 88.89 RSS support pages

11211 2.00 88.89 Filename search

11212 2.00 88.89 Image link search

1272 2.00 88.89 Recent update time

1281 2.00 88.89 Location of searching user [i.e. weather forecast]

1282 2.00 88.89 Location of country

11223 2.00 88.89 Content Controlled Search

134 2.00 88.89 Sport score

135 2.00 88.89 Stock price

136 9.00 88.89 Literature [books,..]

2132 2.00 88.89 Customization of page rank

223 2.00 88.89 Availability of cached result

232 2.00 88.89 Manual user-guide

1123 2.00 88.89 Audio Searching

1212 2.00 88.89 Duplicate Content

1221 2.00 88.89 Adult content filter

413 2.00 88.89 Coverage of user need

11141 2.00 88.89 Search numeric expression in documents

421 90.00 89.90 Accuracy

1113 46.00 90.38 Search with different languages

424 93.00 92.93 Specificity

Yahoo


of Yahoo is Performance.


The weakest subsystem of Yahoo is Reliability.







-------------------------------------------------------------

11142 4.00 0.00 Compute numeric expression

233 28.00 57.50 FAQ


421 75.00 74.75 Accuracy

1244 3.00 75.00 field to search keyword [title,text,URL,Link]




1243 3.00 77.78 Numeric range filter





137 8.00 77.78 Maps

221 8.00 77.78 Result visibility

11164 3.00 77.78 Optional Search (using ‘OR’ operator)




423 78.00 77.78 Recall

422 80.00 79.80 Precision




125 26.00 86.67 File specific filter [pdf, word, excel sheet,..]

bing


of bing is Performance.


The weakest subsystem of bing is Reliability.







-------------------------------------------------------------

233 30.00 62.50 FAQ










423 82.00 81.82 Recall


422 85.00 84.85 Precision

421 85.00 84.85 Accuracy



11111 2.00 88.89 Search with one keyword




1233 2.00 88.89 Linked page


11112 2.00 88.89 Search with group of keywords

11153 2.00 88.89 Stemming Search

11161 2.00 88.89 Including Search (using ‘+’ operator)




131 2.00 88.89 Weather

132 2.00 88.89 Blog

133 2.00 88.89 Movie time

135 2.00 88.89 Stock price


137 9.00 88.89 Maps

211 9.00 88.89 Interface visibility



222 9.00 88.89 Accessibility of results



11121 2.00 88.89 Search exact statement

11122 2.00 88.89 Search parts of the statement

412 2.00 88.89 High rank pages results



11213 2.00 88.89 Adjacent text search



Ask


of Ask is Usability.


The weakest subsystem of Ask is Performance.







-------------------------------------------------------------




233 27.00 55.00 FAQ






137 7.00 66.67 Maps

421 70.00 69.70 Accuracy

altavista


of altavista is Performance.


The weakest subsystem of altavista is Reliability.







-------------------------------------------------------------




411 6.00 44.44 Popular pages results

233 28.00 57.50 FAQ




137 7.00 66.67 Maps

421 75.00 74.75 Accuracy










132 3.00 77.78 Blog

133 3.00 77.78 Movie time

135 3.00 77.78 Stock price










423 78.00 77.78 Recall

9.2 DETAILED EVALUATION RESULTS FOR THE SearchEngine PROJECT

[SearchEngine.lst]

Competitive System(s):

1. Google

2. Yahoo

3. bing

4. Ask

5. altavista

PERFORMANCE VARIABLES:

11111. Search with one keyword

11112. Search with group of keywords

11121. Search exact statement

11122. Search parts of the statement

11123. Ignore stop words

11124. Case sensitive Search

1113. Search with different languages

11141. Search numeric expression in documents

11142. Compute numeric expression

11151. Abbreviation Search

11152. Synonyms Search

11153. Stemming Search

11154. Misspell Corrected Search

11161. Including Search (using ‘+’ operator)

11162. Excluding Search (using ‘-‘ operator)

11163. Combinational Search (using ‘*’ operator)

11164. Optional Search (using ‘OR’ operator)

11211. Filename search

11212. Image link search

11213. Adjacent text search

11221. Popularity based Search

11222. Content based Search

11223. Content Controlled Search

1123. Audio Searching

1211. Pages with Spam, doorway

1212. Duplicate Content

1221. Adult content filter

1222. Casino content filter

1231. Pages related to same content

1232. Pages related to same website

1233. Linked page

1241. RSS support pages

1242. Usage rights

1243. Numeric range filter

1244. field to search keyword [title,text,URL,Link]

125. File specific filter [pdf, word, excel sheet,..]

126. Broken link filter

1271. Page created time

1272. Recent update time

1281. Location of searching user [i.e. weather forecast]

1282. Location of country

131. Weather

132. Blog

133. Movie time

134. Sport score

135. Stock price

136. Literature [books,..]

137. Maps

211. Interface visibility

212. Operability

2131. Customization of page size

2132. Customization of page rank

221. Result visibility

222. Accessibility of results

223. Availability of cached result

231. Online help

232. Manual user-guide

233. FAQ

234. Related tutorial material

311. Load Page Time

312. Automatic search suggestion time

3131. Time to evaluate best result

3132. Time to evaluate top N result

32. Resource Consumption

411. Popular pages results

412. High rank pages results

413. Coverage of user need

421. Accuracy

422. Precision

423. Recall

424. Specificity

PERFORMANCE VARIABLES (INPUT VALUES)

# ID Google Yahoo bing Ask altavista

----------------------------------------------------------------

1 11111 1.000 2.000 2.000 3.000 3.000

2 11112 1.000 2.000 2.000 3.000 3.000

3 11121 1.000 1.000 2.000 2.000 3.000

4 11122 1.000 2.000 2.000 2.000 3.000

5 11123 1.000 1.000 1.000 1.000 1.000

6 11124 1.000 1.000 1.000 1.000 1.000

7 1113 46.000 41.000 42.000 6.000 2.000

8 11141 2.000 2.000 2.000 2.000 2.000

9 11142 30.000 4.000 20.000 6.000 4.000

10 11151 1.000 1.000 1.000 2.000 2.000

11 11152 1.000 1.000 1.000 2.000 2.000

12 11153 1.000 1.000 2.000 2.000 2.000

13 11154 1.000 1.000 1.000 2.000 2.000

14 11161 1.000 2.000 2.000 1.000 2.000

15 11162 3.000 3.000 3.000 3.000 3.000

16 11163 2.000 3.000 3.000 3.000 3.000

17 11164 1.000 3.000 1.000 2.000 2.000

18 11211 2.000 2.000 2.000 3.000 3.000

19 11212 2.000 2.000 2.000 2.000 2.000

20 11213 1.000 1.000 2.000 2.000 2.000

21 11221 1.000 2.000 1.000 2.000 2.000

22 11222 1.000 1.000 1.000 1.000 1.000

23 11223 2.000 2.000 2.000 2.000 2.000

24 1123 2.000 2.000 2.000 2.000 2.000

25 1211 1.000 1.000 1.000 2.000 1.000

26 1212 2.000 3.000 3.000 3.000 2.000

27 1221 2.000 2.000 2.000 2.000 2.000

28 1222 3.000 4.000 3.000 4.000 4.000

29 1231 3.000 3.000 2.000 3.000 3.000

30 1232 1.000 1.000 1.000 1.000 1.000

31 1233 2.000 2.000 2.000 2.000 2.000

32 1241 2.000 3.000 2.000 3.000 3.000

33 1242 1.000 2.000 1.000 2.000 2.000

34 1243 1.000 3.000 3.000 3.000 3.000

35 1244 4.000 3.000 4.000 4.000 3.000

36 125 30.000 26.000 27.000 24.000 23.000

37 126 4.000 3.000 3.000 4.000 3.000

38 1271 4.000 3.000 3.000 4.000 3.000

39 1272 2.000 2.000 2.000 2.000 2.000

40 1281 2.000 3.000 2.000 4.000 4.000

41 1282 2.000 2.000 2.000 3.000 4.000

42 131 1.000 2.000 2.000 3.000 2.000

43 132 1.000 2.000 2.000 3.000 3.000

44 133 3.000 1.000 2.000 3.000 3.000

45 134 2.000 1.000 1.000 2.000 2.000

46 135 2.000 1.000 2.000 2.000 3.000

47 136 9.000 9.000 9.000 8.000 8.000

48 137 10.000 8.000 9.000 7.000 7.000

49 211 10.000 9.000 9.000 9.000 9.000

50 212 10.000 10.000 10.000 9.000 9.000

51 2131 10.000 10.000 10.000 10.000 10.000

52 2132 2.000 2.000 2.000 3.000 3.000

53 221 10.000 8.000 9.000 8.000 8.000

54 222 10.000 9.000 9.000 9.000 9.000

55 223 2.000 2.000 2.000 3.000 2.000

56 231 1.000 1.000 1.000 2.000 1.000

57 232 2.000 2.000 2.000 3.000 2.000

58 233 30.000 28.000 30.000 27.000 28.000

59 234 1.000 1.000 1.000 2.000 2.000

60 311 2.000 2.000 2.000 2.000 2.000

61 312 0.500 0.500 0.500 0.500 0.500

62 3131 3.500 3.500 3.500 4.000 4.500

63 3132 3.000 3.000 3.000 3.500 4.000

64 32 76.000 142.000 55.000 360.000 50.000

65 411 1.000 1.000 1.000 3.000 6.000

66 412 1.000 2.000 2.000 3.000 7.000

67 413 2.000 3.000 3.000 4.000 3.000

68 421 90.000 75.000 85.000 70.000 75.000

69 422 95.000 80.000 85.000 80.000 80.000

70 423 88.000 78.000 82.000 78.000 78.000

71 424 93.000 85.000 88.000 84.000 84.000

ELEMENTARY PREFERENCES [%]


----------------------------------------------------------------

1 11111 100.00 88.89 88.89 77.78 77.78

2 11112 100.00 88.89 88.89 77.78 77.78

3 11121 100.00 100.00 88.89 88.89 77.78

4 11122 100.00 88.89 88.89 88.89 77.78

5 11123 100.00 100.00 100.00 100.00 100.00

6 11124 100.00 100.00 100.00 100.00 100.00

7 1113 90.38 80.77 82.69 10.42 2.08

8 11141 88.89 88.89 88.89 88.89 88.89

9 11142 100.00 0.00 66.67 9.09 0.00

10 11151 100.00 100.00 100.00 88.89 88.89

11 11152 100.00 100.00 100.00 88.89 88.89

12 11153 100.00 100.00 88.89 88.89 88.89

13 11154 100.00 100.00 100.00 88.89 88.89

14 11161 100.00 88.89 88.89 100.00 88.89

15 11162 77.78 77.78 77.78 77.78 77.78

16 11163 88.89 77.78 77.78 77.78 77.78

17 11164 100.00 77.78 100.00 88.89 88.89

18 11211 88.89 88.89 88.89 77.78 77.78

19 11212 88.89 88.89 88.89 88.89 88.89

20 11213 100.00 100.00 88.89 88.89 88.89

21 11221 100.00 88.89 100.00 88.89 88.89

22 11222 100.00 100.00 100.00 100.00 100.00

23 11223 88.89 88.89 88.89 88.89 88.89

24 1123 88.89 88.89 88.89 88.89 88.89

25 1211 100.00 100.00 100.00 88.89 100.00

26 1212 88.89 77.78 77.78 77.78 88.89

27 1221 88.89 88.89 88.89 88.89 88.89

28 1222 77.78 66.67 77.78 66.67 66.67

29 1231 77.78 77.78 88.89 77.78 77.78

30 1232 100.00 100.00 100.00 100.00 100.00

31 1233 88.89 88.89 88.89 88.89 88.89

32 1241 88.89 77.78 88.89 77.78 77.78

33 1242 100.00 88.89 100.00 88.89 88.89

34 1243 100.00 77.78 77.78 77.78 77.78

35 1244 100.00 75.00 100.00 100.00 75.00

36 125 100.00 86.67 90.00 80.00 76.67

37 126 66.67 77.78 77.78 66.67 77.78

38 1271 66.67 77.78 77.78 66.67 77.78

39 1272 88.89 88.89 88.89 88.89 88.89

40 1281 88.89 77.78 88.89 66.67 66.67

41 1282 88.89 88.89 88.89 77.78 66.67

42 131 100.00 88.89 88.89 77.78 88.89

43 132 100.00 88.89 88.89 77.78 77.78

44 133 77.78 100.00 88.89 77.78 77.78

45 134 88.89 100.00 100.00 88.89 88.89

46 135 88.89 100.00 88.89 88.89 77.78

47 136 88.89 88.89 88.89 77.78 77.78

48 137 100.00 77.78 88.89 66.67 66.67

49 211 100.00 88.89 88.89 88.89 88.89

50 212 100.00 100.00 100.00 88.89 88.89

51 2131 100.00 100.00 100.00 100.00 100.00

52 2132 88.89 88.89 88.89 77.78 77.78

53 221 100.00 77.78 88.89 77.78 77.78

54 222 100.00 88.89 88.89 88.89 88.89

55 223 88.89 88.89 88.89 77.78 88.89

56 231 100.00 100.00 100.00 88.89 100.00

57 232 88.89 88.89 88.89 77.78 88.89

58 233 62.50 57.50 62.50 55.00 57.50

59 234 100.00 100.00 100.00 88.89 88.89

60 311 100.00 100.00 100.00 100.00 100.00

61 312 100.00 100.00 100.00 100.00 100.00

62 3131 100.00 100.00 100.00 100.00 96.43

63 3132 86.09 86.09 86.09 82.78 79.47

64 32 88.80 75.60 93.00 32.00 94.00

65 411 100.00 100.00 100.00 77.78 44.44

66 412 100.00 88.89 88.89 77.78 33.33

67 413 88.89 77.78 77.78 66.67 77.78

68 421 89.90 74.75 84.85 69.70 74.75

69 422 94.95 79.80 84.85 79.80 79.80

70 423 87.88 77.78 81.82 77.78 77.78

71 424 92.93 84.85 87.88 83.84 83.84

ALL PREFERENCES [%]


----------------------------------------------------------------

1 11111 100.00 88.89 88.89 77.78 77.78

2 11112 100.00 88.89 88.89 77.78 77.78

3 1111 100.00 88.89 88.89 77.78 77.78

4 11121 100.00 100.00 88.89 88.89 77.78

5 11122 100.00 88.89 88.89 88.89 77.78

6 1000 100.00 92.65 88.89 88.89 77.78

7 11123 100.00 100.00 100.00 100.00 100.00

8 11124 100.00 100.00 100.00 100.00 100.00

9 1001 100.00 100.00 100.00 100.00 100.00

10 1002 100.00 93.62 90.35 90.35 80.69

11 1003 100.00 90.50 89.40 81.82 78.78

12 1113 90.38 80.77 82.69 10.42 2.08

13 11141 88.89 88.89 88.89 88.89 88.89

14 11142 100.00 0.00 66.67 9.09 0.00

15 1114 91.07 61.99 84.25 68.20 61.99

16 11151 100.00 100.00 100.00 88.89 88.89

17 11152 100.00 100.00 100.00 88.89 88.89

18 11153 100.00 100.00 88.89 88.89 88.89

19 11154 100.00 100.00 100.00 88.89 88.89

20 1115 100.00 100.00 97.73 88.89 88.89

21 11161 100.00 88.89 88.89 100.00 88.89

22 11162 77.78 77.78 77.78 77.78 77.78

23 11163 88.89 77.78 77.78 77.78 77.78

24 11164 100.00 77.78 100.00 88.89 88.89

25 1116 95.98 83.81 88.19 92.00 86.05

26 1004 95.49 85.31 90.12 69.68 65.58

27 111 99.29 89.68 89.51 79.87 76.65

28 11211 88.89 88.89 88.89 77.78 77.78

29 11212 88.89 88.89 88.89 88.89 88.89

30 11213 100.00 100.00 88.89 88.89 88.89

31 1121 94.16 94.16 88.89 86.45 86.45

32 11221 100.00 88.89 100.00 88.89 88.89

33 11222 100.00 100.00 100.00 100.00 100.00

34 1005 100.00 94.16 100.00 94.16 94.16

35 11223 88.89 88.89 88.89 88.89 88.89

36 1122 97.96 93.20 97.96 93.20 93.20

37 1123 88.89 88.89 88.89 88.89 88.89

38 112 93.97 92.15 92.34 89.80 89.80

39 1211 100.00 100.00 100.00 88.89 100.00

40 1212 88.89 77.78 77.78 77.78 88.89

41 121 94.94 89.74 89.74 83.82 94.94

42 1221 88.89 88.89 88.89 88.89 88.89

43 1222 77.78 66.67 77.78 66.67 66.67

44 122 87.66 86.12 87.66 86.12 86.12

45 1231 77.78 77.78 88.89 77.78 77.78

46 1232 100.00 100.00 100.00 100.00 100.00

47 1233 88.89 88.89 88.89 88.89 88.89

48 123 87.54 87.54 92.71 87.54 87.54

49 1241 88.89 77.78 88.89 77.78 77.78

50 1242 100.00 88.89 100.00 88.89 88.89

51 1243 100.00 77.78 77.78 77.78 77.78

52 1244 100.00 75.00 100.00 100.00 75.00

53 1006 96.05 81.04 88.14 81.04 81.04

54 124 98.19 77.62 94.34 90.62 77.62

55 1007 90.86 85.95 90.38 86.53 87.11

56 125 100.00 86.67 90.00 80.00 76.67

57 126 66.67 77.78 77.78 66.67 77.78

58 1271 66.67 77.78 77.78 66.67 77.78

59 1272 88.89 88.89 88.89 88.89 88.89

60 127 84.25 86.62 86.62 84.25 86.62

61 1281 88.89 77.78 88.89 66.67 66.67

62 1282 88.89 88.89 88.89 77.78 66.67

63 128 88.89 85.50 88.89 74.38 66.67

64 1008 86.73 84.98 86.75 77.74 77.27

65 12 89.35 85.61 89.07 83.16 83.30

66 131 100.00 88.89 88.89 77.78 88.89

67 132 100.00 88.89 88.89 77.78 77.78

68 133 77.78 100.00 88.89 77.78 77.78

69 134 88.89 100.00 100.00 88.89 88.89

70 135 88.89 100.00 88.89 88.89 77.78

71 136 88.89 88.89 88.89 77.78 77.78

72 137 100.00 77.78 88.89 66.67 66.67

73 1009 90.83 93.27 88.89 77.78 81.59

74 1010 92.71 89.21 91.06 78.06 75.92

75 13 92.05 90.62 90.30 77.96 77.89

76 11 97.10 90.65 90.62 83.61 81.49

77 1011 94.26 88.83 90.07 83.45 82.12

78 1 93.79 89.21 90.12 82.26 81.20

79 211 100.00 88.89 88.89 88.89 88.89

80 212 100.00 100.00 100.00 88.89 88.89

81 1012 100.00 95.13 95.13 88.89 88.89

82 2131 100.00 100.00 100.00 100.00 100.00

83 2132 88.89 88.89 88.89 77.78 77.78

84 213 91.07 91.07 91.07 82.05 82.05

85 21 98.81 94.59 94.59 87.98 87.98

86 221 100.00 77.78 88.89 77.78 77.78

87 222 100.00 88.89 88.89 88.89 88.89

88 223 88.89 88.89 88.89 77.78 88.89

89 1013 97.74 88.89 88.89 86.62 88.89

90 231 100.00 100.00 100.00 88.89 100.00

91 232 88.89 88.89 88.89 77.78 88.89

92 233 62.50 57.50 62.50 55.00 57.50

93 234 100.00 100.00 100.00 88.89 88.89

94 23 88.60 87.41 88.60 78.25 84.22

95 1014 92.71 85.78 88.72 79.83 83.86

96 2 96.95 88.40 88.86 85.55 88.10

97 311 100.00 100.00 100.00 100.00 100.00

98 312 100.00 100.00 100.00 100.00 100.00

99 3131 100.00 100.00 100.00 100.00 96.43

100 3132 86.09 86.09 86.09 82.78 79.47

101 1015 100.00 100.00 100.00 100.00 100.00

102 1016 97.16 97.16 97.16 96.46 92.94

103 31 99.13 99.13 99.13 98.91 97.79

104 32 88.80 75.60 93.00 32.00 94.00

105 3 95.82 90.87 97.22 63.49 96.62

106 411 100.00 100.00 100.00 77.78 44.44

107 412 100.00 88.89 88.89 77.78 33.33

108 413 88.89 77.78 77.78 66.67 77.78

109 41 94.37 86.44 86.44 72.13 57.47

110 421 89.90 74.75 84.85 69.70 74.75

111 422 94.95 79.80 84.85 79.80 79.80

112 423 87.88 77.78 81.82 77.78 77.78

113 424 92.93 84.85 87.88 83.84 83.84

114 42 91.90 79.86 85.14 78.51 79.57

115 4 93.87 85.05 86.18 73.33 60.97

116 9999 94.89 88.18 89.78 77.35 78.42

Elementary Preferences for System = Google


-------------------------------------------------------------





11123 1.00 100.00 Ignore stop words

11124 1.00 100.00 Case sensitive Search




11151 1.00 100.00 Abbreviation Search

11152 1.00 100.00 Synonyms Search


11154 1.00 100.00 Misspell Corrected Search








11221 1.00 100.00 Popularity based Search

11222 1.00 100.00 Content based Search



1211 1.00 100.00 Pages with Spam, doorway





1232 1.00 100.00 Pages related to same website

1233 2.00 88.89 Linked page


1242 1.00 100.00 Usage rights









131 1.00 100.00 Weather

132 1.00 100.00 Blog

133 3.00 77.78 Movie time

134 2.00 88.89 Sport score

135 2.00 88.89 Stock price


137 10.00 100.00 Maps


212 10.00 100.00 Operability

2131 10.00 100.00 Customization of page size





231 1.00 100.00 Online help


233 30.00 62.50 FAQ

234 1.00 100.00 Related tutorial material

311 2.00 100.00 Load Page Time

312 0.50 100.00 Automatic search suggestion time

3131 3.50 100.00 Time to evaluate best result






421 90.00 89.90 Accuracy

422 95.00 94.95 Precision

423 88.00 87.88 Recall


Sorted Elementary Preferences for System = Google


-------------------------------------------------------------

233 30.00 62.50 FAQ





133 3.00 77.78 Movie time



423 88.00 87.88 Recall



1233 2.00 88.89 Linked page








134 2.00 88.89 Sport score

135 2.00 88.89 Stock price










421 90.00 89.90 Accuracy



422 95.00 94.95 Precision








131 1.00 100.00 Weather

132 1.00 100.00 Blog





137 10.00 100.00 Maps


212 10.00 100.00 Operability






231 1.00 100.00 Online help














1242 1.00 100.00 Usage rights


Elementary Preferences for System = Yahoo


-------------------------------------------------------------































1233 2.00 88.89 Linked page


1242 2.00 88.89 Usage rights









131 2.00 88.89 Weather

132 2.00 88.89 Blog

133 1.00 100.00 Movie time

134 1.00 100.00 Sport score

135 1.00 100.00 Stock price


137 8.00 77.78 Maps


212 10.00 100.00 Operability






231 1.00 100.00 Online help


233 28.00 57.50 FAQ










421 75.00 74.75 Accuracy

422 80.00 79.80 Precision

423 78.00 77.78 Recall


Sorted Elementary Preferences for System = Yahoo


-------------------------------------------------------------


233 28.00 57.50 FAQ


421 75.00 74.75 Accuracy










137 8.00 77.78 Maps






423 78.00 77.78 Recall

422 80.00 79.80 Precision










131 2.00 88.89 Weather

132 2.00 88.89 Blog











1233 2.00 88.89 Linked page



1242 2.00 88.89 Usage rights




212 10.00 100.00 Operability






231 1.00 100.00 Online help










133 1.00 100.00 Movie time

134 1.00 100.00 Sport score

135 1.00 100.00 Stock price




Elementary Preferences for System = bing


-------------------------------------------------------------































1233 2.00 88.89 Linked page


1242 1.00 100.00 Usage rights









131 2.00 88.89 Weather

132 2.00 88.89 Blog

133 2.00 88.89 Movie time

134 1.00 100.00 Sport score

135 2.00 88.89 Stock price


137 9.00 88.89 Maps


212 10.00 100.00 Operability






231 1.00 100.00 Online help


233 30.00 62.50 FAQ










421 85.00 84.85 Accuracy

422 85.00 84.85 Precision

423 82.00 81.82 Recall


Sorted Elementary Preferences for System = bing


-------------------------------------------------------------

233 30.00 62.50 FAQ










423 82.00 81.82 Recall


422 85.00 84.85 Precision

421 85.00 84.85 Accuracy







1233 2.00 88.89 Linked page








131 2.00 88.89 Weather

132 2.00 88.89 Blog

133 2.00 88.89 Movie time

135 2.00 88.89 Stock price


137 9.00 88.89 Maps






















231 1.00 100.00 Online help










134 1.00 100.00 Sport score

1242 1.00 100.00 Usage rights




212 10.00 100.00 Operability

Elementary Preferences for System = Ask


-------------------------------------------------------------































1233 2.00 88.89 Linked page


1242 2.00 88.89 Usage rights









131 3.00 77.78 Weather

132 3.00 77.78 Blog

133 3.00 77.78 Movie time

134 2.00 88.89 Sport score

135 2.00 88.89 Stock price


137 7.00 66.67 Maps


212 9.00 88.89 Operability






231 2.00 88.89 Online help


233 27.00 55.00 FAQ










421 70.00 69.70 Accuracy

422 80.00 79.80 Precision

423 78.00 77.78 Recall


Sorted Elementary Preferences for System = Ask


-------------------------------------------------------------




233 27.00 55.00 FAQ






137 7.00 66.67 Maps

421 70.00 69.70 Accuracy






131 3.00 77.78 Weather

132 3.00 77.78 Blog

133 3.00 77.78 Movie time













423 78.00 77.78 Recall

422 80.00 79.80 Precision






134 2.00 88.89 Sport score

135 2.00 88.89 Stock price









231 2.00 88.89 Online help

1233 2.00 88.89 Linked page



1242 2.00 88.89 Usage rights



















Elementary Preferences for System = altavista


-------------------------------------------------------------































1233 2.00 88.89 Linked page


1242 2.00 88.89 Usage rights









131 2.00 88.89 Weather

132 3.00 77.78 Blog

133 3.00 77.78 Movie time

134 2.00 88.89 Sport score

135 3.00 77.78 Stock price


137 7.00 66.67 Maps








231 1.00 100.00 Online help


233 28.00 57.50 FAQ










421 75.00 74.75 Accuracy

422 80.00 79.80 Precision

423 78.00 77.78 Recall


Sorted Elementary Preferences for System = altavista


-------------------------------------------------------------





233 28.00 57.50 FAQ




137 7.00 66.67 Maps

421 75.00 74.75 Accuracy










132 3.00 77.78 Blog

133 3.00 77.78 Movie time

135 3.00 77.78 Stock price










423 78.00 77.78 Recall


422 80.00 79.80 Precision



131 2.00 88.89 Weather



134 2.00 88.89 Sport score







1233 2.00 88.89 Linked page






1242 2.00 88.89 Usage rights












231 1.00 100.00 Online help







Subsystem Preferences for System = Google

ID E[%] Subsystem

-------------------------------------------------------------

1111 100.00 Keyword Searching

1000 100.00 Temp GCD for 11121 11122

1001 100.00 Temp GCD for 11123 11124

1002 100.00 Temp GCD for 1000 1001

1003 100.00 Temp GCD for 1111 1002

1114 91.07 Numeric Expression Search

1115 100.00 Relational Searching

1116 95.98 Operator Searching

1004 95.49 Temp GCD for 1113 1114 1115 1116

111 99.29 Text Searching

1121 94.16 Image Searching

1005 100.00 * Temp GCD for 11221 11222

1122 97.96 Video Searching

112 93.97 Multimedia Search

121 94.94 Security filter

122 87.66 Citation filter

123 87.54 Domain filter

1006 96.05 * 1006

124 98.19 Extended filter

1007 90.86 *Temp GCD for 122 123 124

127 84.25 Time filter

128 88.89 Location filter

1008 86.73 *Temp GCD for 125 126 127 128

12 89.35 Searching Filter

1009 90.83 Temp GCD for 131 132 133

1010 92.71 Temp GCD for 134 135 136 137

13 92.05 Specific activity Search

11 97.10 Searching Input

1011 94.26 Temp GCD for 11 12

1 93.79 Functionality

1012 100.00 Temp GCD for 211 212

213 91.07 Customization

21 98.81 Interface usability

1013 97.74 Temp GCD for 222 223

23 88.60 User guide

1014 92.71 Temp GCD for 221 1013 23

2 96.95 Usability

1015 100.00 Temp GCD for 311 312

1016 97.16 3131 3132

31 99.13 21

3 95.82 Performance

41 94.37 User satisfaction

42 91.90 Confusion matrix

4 93.87 Reliability

9999 94.89 Whole_Search_Engine

Sorted Subsystem Preferences for System = Google

ID E[%] Subsystem

-------------------------------------------------------------


1008 86.73 *Temp GCD for 125 126 127 128



23 88.60 User guide



1009 90.83 Temp GCD for 131 132 133

1007 90.86 *Temp GCD for 122 123 124





1014 92.71 Temp GCD for 221 1013 23

1010 92.71 Temp GCD for 134 135 136 137


4 93.87 Reliability



1011 94.26 Temp GCD for 11 12




1004 95.49 Temp GCD for 1113 1114 1115 1116

3 95.82 Performance


1006 96.05 * 1006

2 96.95 Usability


1016 97.16 3131 3132

1013 97.74 Temp GCD for 222 223




31 99.13 21


1000 100.00 Temp GCD for 11121 11122

1015 100.00 Temp GCD for 311 312

1001 100.00 Temp GCD for 11123 11124

1012 100.00 Temp GCD for 211 212

1002 100.00 Temp GCD for 1000 1001

1003 100.00 Temp GCD for 1111 1002

1005 100.00 * Temp GCD for 11221 11222



Subsystem Preferences for System = Yahoo

ID E[%] Subsystem

-------------------------------------------------------------


1000 92.65 Temp GCD for 11121 11122

1001 100.00 Temp GCD for 11123 11124

1002 93.62 Temp GCD for 1000 1001

1003 90.50 Temp GCD for 1111 1002




1004 85.31 Temp GCD for 1113 1114 1115 1116



1005 94.16 * Temp GCD for 11221 11222






1006 81.04 * 1006


1007 85.95 *Temp GCD for 122 123 124



1008 84.98 *Temp GCD for 125 126 127 128


1009 93.27 Temp GCD for 131 132 133

1010 89.21 Temp GCD for 134 135 136 137



1011 88.83 Temp GCD for 11 12


1012 95.13 Temp GCD for 211 212



1013 88.89 Temp GCD for 222 223

23 87.41 User guide

1014 85.78 Temp GCD for 221 1013 23

2 88.40 Usability

1015 100.00 Temp GCD for 311 312

1016 97.16 3131 3132

31 99.13 21

3 90.87 Performance



4 85.05 Reliability


Sorted Subsystem Preferences for System = Yahoo

ID E[%] Subsystem

-------------------------------------------------------------




1006 81.04 * 1006


1008 84.98 *Temp GCD for 125 126 127 128

4 85.05 Reliability

1004 85.31 Temp GCD for 1113 1114 1115 1116



1014 85.78 Temp GCD for 221 1013 23

1007 85.95 *Temp GCD for 122 123 124




23 87.41 User guide



2 88.40 Usability

1011 88.83 Temp GCD for 11 12

1013 88.89 Temp GCD for 222 223


1010 89.21 Temp GCD for 134 135 136 137




1003 90.50 Temp GCD for 1111 1002



3 90.87 Performance



1000 92.65 Temp GCD for 11121 11122


1009 93.27 Temp GCD for 131 132 133

1002 93.62 Temp GCD for 1000 1001


1005 94.16 * Temp GCD for 11221 11222


1012 95.13 Temp GCD for 211 212

1016 97.16 3131 3132

31 99.13 21


1015 100.00 Temp GCD for 311 312

1001 100.00 Temp GCD for 11123 11124

Subsystem Preferences for System = bing

ID E[%] Subsystem

-------------------------------------------------------------


1000 88.89 Temp GCD for 11121 11122

1001 100.00 Temp GCD for 11123 11124

1002 90.35 Temp GCD for 1000 1001

1003 89.40 Temp GCD for 1111 1002




1004 90.12 Temp GCD for 1113 1114 1115 1116



1005 100.00 * Temp GCD for 11221 11222






1006 88.14 * 1006


1007 90.38 *Temp GCD for 122 123 124



1008 86.75 *Temp GCD for 125 126 127 128


1009 88.89 Temp GCD for 131 132 133

1010 91.06 Temp GCD for 134 135 136 137



1011 90.07 Temp GCD for 11 12


1012 95.13 Temp GCD for 211 212



1013 88.89 Temp GCD for 222 223

23 88.60 User guide

1014 88.72 Temp GCD for 221 1013 23

2 88.86 Usability

1015 100.00 Temp GCD for 311 312

1016 97.16 3131 3132

31 99.13 21

3 97.22 Performance



4 86.18 Reliability


Sorted Subsystem Preferences for System = bing

ID E[%] Subsystem

-------------------------------------------------------------



4 86.18 Reliability



1008 86.75 *Temp GCD for 125 126 127 128


1006 88.14 * 1006


23 88.60 User guide

1014 88.72 Temp GCD for 221 1013 23

2 88.86 Usability

1000 88.89 Temp GCD for 11121 11122




1009 88.89 Temp GCD for 131 132 133

1013 88.89 Temp GCD for 222 223


1003 89.40 Temp GCD for 1111 1002




1011 90.07 Temp GCD for 11 12

1004 90.12 Temp GCD for 1113 1114 1115 1116



1002 90.35 Temp GCD for 1000 1001

1007 90.38 *Temp GCD for 122 123 124


1010 91.06 Temp GCD for 134 135 136 137






1012 95.13 Temp GCD for 211 212

1016 97.16 3131 3132

3 97.22 Performance



31 99.13 21

1001 100.00 Temp GCD for 11123 11124

1015 100.00 Temp GCD for 311 312

1005 100.00 * Temp GCD for 11221 11222

Subsystem Preferences for System = Ask

ID E[%] Subsystem

-------------------------------------------------------------


1000 88.89 Temp GCD for 11121 11122

1001 100.00 Temp GCD for 11123 11124

1002 90.35 Temp GCD for 1000 1001

1003 81.82 Temp GCD for 1111 1002




1004 69.68 Temp GCD for 1113 1114 1115 1116



1005 94.16 * Temp GCD for 11221 11222






1006 81.04 * 1006


1007 86.53 *Temp GCD for 122 123 124



1008 77.74 *Temp GCD for 125 126 127 128


1009 77.78 Temp GCD for 131 132 133

1010 78.06 Temp GCD for 134 135 136 137



1011 83.45 Temp GCD for 11 12


1012 88.89 Temp GCD for 211 212



1013 86.62 Temp GCD for 222 223

23 78.25 User guide

1014 79.83 Temp GCD for 221 1013 23

2 85.55 Usability

1015 100.00 Temp GCD for 311 312

1016 96.46 3131 3132

31 98.91 21

3 63.49 Performance



4 73.33 Reliability


Sorted Subsystem Preferences for System = Ask

ID E[%] Subsystem

-------------------------------------------------------------

3 63.49 Performance


1004 69.68 Temp GCD for 1113 1114 1115 1116


4 73.33 Reliability



1008 77.74 *Temp GCD for 125 126 127 128


1009 77.78 Temp GCD for 131 132 133


1010 78.06 Temp GCD for 134 135 136 137

23 78.25 User guide


1014 79.83 Temp GCD for 221 1013 23


1006 81.04 * 1006

1003 81.82 Temp GCD for 1111 1002




1011 83.45 Temp GCD for 11 12




2 85.55 Usability



1007 86.53 *Temp GCD for 122 123 124

1013 86.62 Temp GCD for 222 223



1012 88.89 Temp GCD for 211 212

1000 88.89 Temp GCD for 11121 11122



1002 90.35 Temp GCD for 1000 1001




1005 94.16 * Temp GCD for 11221 11222

1016 96.46 3131 3132

31 98.91 21

1015 100.00 Temp GCD for 311 312

1001 100.00 Temp GCD for 11123 11124

Subsystem Preferences for System = altavista

ID E[%] Subsystem

-------------------------------------------------------------


1000 77.78 Temp GCD for 11121 11122

1001 100.00 Temp GCD for 11123 11124

1002 80.69 Temp GCD for 1000 1001

1003 78.78 Temp GCD for 1111 1002




1004 65.58 Temp GCD for 1113 1114 1115 1116



1005 94.16 * Temp GCD for 11221 11222






1006 81.04 * 1006


1007 87.11 *Temp GCD for 122 123 124



1008 77.27 *Temp GCD for 125 126 127 128


1009 81.59 Temp GCD for 131 132 133

1010 75.92 Temp GCD for 134 135 136 137



1011 82.12 Temp GCD for 11 12


1012 88.89 Temp GCD for 211 212



1013 88.89 Temp GCD for 222 223

23 84.22 User guide

1014 83.86 Temp GCD for 221 1013 23

2 88.10 Usability

1015 100.00 Temp GCD for 311 312

1016 92.94 3131 3132

31 97.79 21

3 96.62 Performance



4 60.97 Reliability


Sorted Subsystem Preferences for System = altavista

ID E[%] Subsystem

-------------------------------------------------------------


4 60.97 Reliability


1004 65.58 Temp GCD for 1113 1114 1115 1116


1010 75.92 Temp GCD for 134 135 136 137


1008 77.27 *Temp GCD for 125 126 127 128


1000 77.78 Temp GCD for 11121 11122




1003 78.78 Temp GCD for 1111 1002


1002 80.69 Temp GCD for 1000 1001

1006 81.04 * 1006



1009 81.59 Temp GCD for 131 132 133


1011 82.12 Temp GCD for 11 12


1014 83.86 Temp GCD for 221 1013 23

23 84.22 User guide





1007 87.11 *Temp GCD for 122 123 124



2 88.10 Usability

1012 88.89 Temp GCD for 211 212

1013 88.89 Temp GCD for 222 223



1016 92.94 3131 3132


1005 94.16 * Temp GCD for 11221 11222


3 96.62 Performance

31 97.79 21

1015 100.00 Temp GCD for 311 312

1001 100.00 Temp GCD for 11123 11124

Note: All sorted reports are divided in two parts: the first part contains

components that are below the average quality level for the analyzed system

and the second part contains those components that are above the average

quality level

9.3 SearchEngine.res

Google

100 100 100 100 100 100 100 100 100 100

100 90.3846 88.8889 100 91.0711 100 100 100 100 100

100 77.7778 88.8889 100 95.978 95.4867 99.2907 88.8889 88.8889 100

94.1613 100 100 100 88.8889 97.959 88.8889 93.967 100 88.8889

94.9383 88.8889 77.7778 87.6579 77.7778 100 88.8889 87.5421 88.8889 100

100 100 96.0476 98.1875 90.8575 100 66.6667 66.6667 88.8889 84.2513

88.8889 88.8889 88.8889 86.7288 89.3539 100 100 77.7778 88.8889 88.8889

88.8889 100 90.8257 92.7102 92.0489 97.1004 94.2619 93.7855 100 100

100 100 88.8889 91.0711 98.8115 100 100 88.8889 97.738 100

88.8889 62.5 100 88.5995 92.7073 96.9464 100 100 100 86.0927

100 97.1553 99.1317 88.8 95.8232 100 100 88.8889 94.3746 89.899

94.9495 87.8788 92.9293 91.9015 93.8709 94.8858

Yahoo

88.8889 88.8889 88.8889 100 88.8889 92.6521 100 100 100 93.6205

90.4972 80.7692 88.8889 0 61.9853 100 100 100 100 100

88.8889 77.7778 77.7778 77.7778 83.8056 85.312 89.6803 88.8889 88.8889 100

94.1613 88.8889 100 94.1633 88.8889 93.1976 88.8889 92.1478 100 77.7778

89.7373 88.8889 66.6667 86.1237 77.7778 100 88.8889 87.5421 77.7778 88.8889

77.7778 75 81.0445 77.6207 85.9511 86.6667 77.7778 77.7778 88.8889 86.6216

77.7778 88.8889 85.4964 84.9819 85.6086 88.8889 88.8889 100 100 100

88.8889 77.7778 93.2663 89.2081 90.6206 90.6536 88.8309 89.214 88.8889 100

95.1324 100 88.8889 91.0711 94.5942 77.7778 88.8889 88.8889 88.8889 100

88.8889 57.5 100 87.4093 85.7789 88.4011 100 100 100 86.0927

100 97.1553 99.1317 75.6 90.8727 100 88.8889 77.7778 86.4394 74.7475

79.798 77.7778 84.8485 79.8605 85.0506 88.18

bing

88.8889 88.8889 88.8889 88.8889 88.8889 88.8889 100 100 100 90.3518

89.3963 82.6923 88.8889 66.6667 84.2513 100 100 88.8889 100 97.7303

88.8889 77.7778 77.7778 100 88.1911 90.1191 89.5087 88.8889 88.8889 88.8889

88.8889 100 100 100 88.8889 97.959 88.8889 92.3353 100 77.7778

89.7373 88.8889 77.7778 87.6579 88.8889 100 88.8889 92.7142 88.8889 100

77.7778 100 88.1402 94.3431 90.384 90 77.7778 77.7778 88.8889 86.6216

88.8889 88.8889 88.8889 86.7543 89.0683 88.8889 88.8889 88.8889 100 88.8889

88.8889 88.8889 88.8889 91.0635 90.3001 90.6213 90.0725 90.1213 88.8889 100

95.1324 100 88.8889 91.0711 94.5942 88.8889 88.8889 88.8889 88.8889 100

88.8889 62.5 100 88.5995 88.7152 88.8618 100 100 100 86.0927

100 97.1553 99.1317 93 97.2208 100 88.8889 77.7778 86.4394 84.8485

84.8485 81.8182 87.8788 85.1395 86.1767 89.7764

Ask

77.7778 77.7778 77.7778 88.8889 88.8889 88.8889 100 100 100 90.3518

81.8171 10.4167 88.8889 9.09091 68.2024 88.8889 88.8889 88.8889 88.8889 88.8889

100 77.7778 77.7778 88.8889 91.9989 69.6791 79.8701 77.7778 88.8889 88.8889

86.4543 88.8889 100 94.1633 88.8889 93.1976 88.8889 89.803 88.8889 77.7778

83.819 88.8889 66.6667 86.1237 77.7778 100 88.8889 87.5421 77.7778 88.8889

77.7778 100 81.0445 90.6169 86.5303 80 66.6667 66.6667 88.8889 84.2513

66.6667 77.7778 74.3761 77.7405 83.1634 77.7778 77.7778 77.7778 88.8889 88.8889

77.7778 66.6667 77.7778 78.0629 77.9631 83.6054 83.4502 82.2622 88.8889 88.8889

88.8889 100 77.7778 82.0512 87.9798 77.7778 88.8889 77.7778 86.6216 88.8889

77.7778 55 88.8889 78.2542 79.8324 85.5476 100 100 100 82.7815

100 96.4576 98.9141 32 63.4887 77.7778 77.7778 66.6667 72.1309 69.697

79.798 77.7778 83.8384 78.5127 73.3343 77.3459

altavista

77.7778 77.7778 77.7778 77.7778 77.7778 77.7778 100 100 100 80.6944

78.7777 2.08333 88.8889 0 61.9853 88.8889 88.8889 88.8889 88.8889 88.8889

88.8889 77.7778 77.7778 88.8889 86.0481 65.5789 76.6524 77.7778 88.8889 88.8889

86.4543 88.8889 100 94.1633 88.8889 93.1976 88.8889 89.803 100 88.8889

94.9383 88.8889 66.6667 86.1237 77.7778 100 88.8889 87.5421 77.7778 88.8889

77.7778 75 81.0445 77.6207 87.1146 76.6667 77.7778 77.7778 88.8889 86.6216

66.6667 66.6667 66.6667 77.2685 83.3033 88.8889 77.7778 77.7778 88.8889 77.7778

77.7778 66.6667 81.5946 75.9182 77.8871 81.4889 82.1161 81.2023 88.8889 88.8889

88.8889 100 77.7778 82.0512 87.9798 77.7778 88.8889 88.8889 88.8889 100

88.8889 57.5 88.8889 84.2168 83.8634 88.0973 100 100 96.4286 79.4702

100 92.9375 97.7867 94 96.6235 44.4444 33.3333 77.7778 57.4659 74.7475

79.798 77.7778 83.8384 79.5652 60.9744 78.4239

9.4 SUMMARY OF RESULTS FOR THE SearchEngine PROJECT

[SearchEngine.sum]

This report presents the final evaluation results (global preferences)

for the SearchEngine project. The basic parameters of this

project are:

Number of competitive systems: n = 5

Global preference range: [Emin, Emax] = [77.35%, 94.89%]

Following are the final numeric results. For more detailed analysis please

see other LSP reports (SearchEngine.lst and SearchEngine.txt).

SORTED GLOBAL PREFERENCES

1. 94.89% Google

2. 89.78% bing

3. 88.18% Yahoo

4. 78.42% altavista

5. 77.35% Ask

SORTED NORMALIZED GLOBAL PREFERENCES

1. 100.00% Google

2. 94.62% bing

3. 92.93% Yahoo

4. 82.65% altavista

5. 81.51% Ask

Interpretation of results:

For the majority of standard criteria it is reasonable to request that

competitive systems satisfy at least 2/3 of requirements. If E<67% this

means either that the quality of the evaluated system is not sufficient,

or that the selected criterion specifies unrealistically high requirements

that cannot be sufficiently satisfied.

A reliability analysis is necessary to compute the level of confidence

that one system is better than all other systems. Small difference between

two global preferences (e.g. 3%) can be sufficiently significant if one

system outperforms others in the majority of elementary preferences.

An indicator of dominance of system A over system B can be defined as a

percentage of inputs where A is better than B.

Normalized results are convenient for relative comparison of analyzed

systems. Normalization is performed so that the maximum global preference,

the maximum global quality (Q), and the minimum global cost are interpreted

as 100%. Therefore, the preference of 85% is interpreted as 15% below the

best result. The normalized cost of 120% indicates the cost that is 20%

above the minimum cost. However, the relative comparison makes sense only

if absolute preferences are sufficiently high and absolute costs are in

the acceptable range.

Documents

Rajan Dhabalia LSP Research Paper