Why we need an independent index of the Web

Dirk Lewandowski dirk.lewandowski@haw-hamburg.de http://www.bui.haw-hamburg.de/lewandowski.html @Dirk_Lew Society of the Query Conference, Amsterdam, 7/11/2013

The “local copy” of the Web

•  Web Indexing –  New, changed, deleted document

–  “Holy grail” of keeping the index complete and current

Risvik, K. M., & Michelsen, R. (2002). Search engines and web dynamics. Computer Networks, 39(3), 289–302.

Representation of documents in a search engine

Referring documents à Document à Metadata (examplex)

heading1

heading 2

Anchor text

Anchor text

Anchor text

From the source code - Title - Description - Keywords - Author

From the document (document info) - Length - Date - Decay -  Name of the author

From the Web - PageRank - Number of citations

The User’s Perspective

•  Everyone uses search engines (Purcell, Brenner & Raine, 2012; van Eimeren & Frees, 2012)

•  Market is dominated by Google (ComScore data) •  Users rely on

–  Google’s method of ordering results

–  Google’s method of collecting data

à If Google hasn’t seen it — and indexed it — or kept it up to date, it can’t be found with a search query.

Freshness of Web search engines (see Lewandowski, Wahlig & Meyer-Bautor, 2006; Lewandowski, 2008)

Original (as of yesterday) Google‘s copy (as of yesterday)

What about the alternatives to Google?

•  Many “seems to be” search engines –  Accessing the data of another search engine

–  Representing nothing more than an alternative user interface to one of the more well-known engines

–  In many cases, that turns out to be Google

–  E.g., in Germany, we can see that the major internet portals T-Online, GMX, AOL, and web.de all display results obtained from Google

Why is one search engine not enough?

•  We need more than one search engine to ensure that a broad range of opinions are represented in the search market.

•  Users should have the choice between different worldviews which originate as a product of algorithm-based search result generation

•  Ideology-free search algorithms are simply not possible

Alternative Search Engine Indexes

•  There are only a handful of search engines that operate their own indexes, due to costs and technical complexity

•  Search engines start-ups –  Use an existing external index –  Focus on a specialised topic (which requires only a small index)

–  Aggregate data from different search engines (meta search engine)

•  Actual search engine startups like Blekko and Duck Duck Go are more the exception than the rule

Partner model

•  “Real” search engine providers such as Google and Bing operate their own search engines but also provide their search results to partners

•  All the major web portals have now embraced this model.

•  Income through ads; revenue-sharing

•  Attractiveness of the model –  The search engine provider encounters only minimal costs

–  The operator of the portal no longer needs to go to the great expense of running its own search engine.

–  The partner index model has served to thin out the competition in the search industry.

Access to Search Engine Indexes

•  Application programming interfaces (APIs) –  No direct access to the search engine index –  Limited number of top results which have already been ranked by the search

engine provider

–  Access via APIs is similar to what is occurring at the meta-search engines

–  The representation of the document in the source search engine is also not included

Alternative Search Engines

•  What constitutes an “alternative search engine”? –  All search engines that are not Google? (“Google Killers“, e.g., Cuil)

–  Some alternatives are not perceived as such because they are considered to be simply the same as Google (e.g., Bing)

–  Search engines which explicitly position themselves as an alternative to Google through a regional approach (e.g., Seekport)

–  New approaches to search / “Real alternatives”: Alternative approaches to gathering and representing web content

Public Support for Search Engine Technology?

•  Quaero/Theseus: Funding a “Google Killer”? –  Quaero: Technologies for multimedia searching.

–  Theseus: Semantic technologies for business-to-business applications (without focusing exclusively on search).

•  The proposal to provide government funding for search engine technology has been subject to intense criticism in the past

•  Establish a single alternative?

•  A number of factors which would cause it to fail –  Poor marketing –  Graphic design of the user interface –  ...

•  Regardless of the reason, a failure of the new search engine would result in the entire publicly funded initiative failing.

Economic perspective

•  Only the largest internet companies are able to afford large indexes. •  Microsoft is the only company besides Google to possess a comprehensive

search engine index.

•  Yahoo gave up on its own index several years ago

•  It appears as though operating a dedicated index is attractive to practically no one — and there are hardly any candidates with the necessary financial resources in any case

The Solution

•  Create the conditions that will make establishing alternative search engines possible

•  We can expect that the possibilities it presents would benefit a number of different companies, individuals, and institutions.

•  The result will be fair competition to develop the best concepts for using the data provided by the index.

Vision

•  “An index of the web that can be accessed at fair conditions for everyone”

–  “Everyone” means that anyone who is interested can access the index.

–  “Fair conditions” does not mean that access to the index must be free of charge for everyone. A certain number of document requests per day should be available at no cost in order to promote non-profit projects.

–  “Access” to the index can be defined as the ability to automatically query the index with ease.

–  The concept “index of the web” is intended to cover as much of the web as possible

Funding and operation

•  Funding –  This type of project cannot be supported by any one country alone. The only

feasible option is a pan-European initiative.

•  Who would operate the index? –  Existing research institution or newly-founded institution –  The operator of the index should not obtain the exclusive right to determine the

way in which the documents are used or made available (à Board of trustees)

Conclusion: Advantages of an independent index of the web

•  Motivate companies, institutions, and developers pursuing personal projects to create their own search applications.

•  The data available on the web is so boundless that it lends itself to countless applications in a broad range of fields.

•  Enable applications we are not yet capable of even imagining.

•  An open structure, transparency with respect to access, and the assurance of permanent availability thanks to state sponsorship would lay the groundwork for innovation.

Thank you

Prof. Dr. Dirk Lewandowski Hochschule für Angewandte Wissenschaften Hamburg dirk.lewandowski@haw-hamburg,de Twitter: Dirk_Lew http://www.bui.haw-hamburg.de/lewandowski.html http://www.searchstudies.org