View
215
Download
0
Category
Tags:
Preview:
Citation preview
1
Web Search EnvironmentsWeb Crawling Metadata using RDF and Dublin Core
Dave Becketthttp://purl.org/net/dajobe/
Slides:http://ilrt.org/people/cmdjb/talks/tnc2002/
2
Introduction
• Overview of SGs and Web Crawling• Why WSE, what’s new? Novel
results• Future work (or stuff we didn’t do)
and conclusions
3
Overview
• Digital Library community• In UK, subject-specific gateways (SGs)• Want to improve: scope (more),
timeliness (fresh), cost (less)• Stay professional – the Quality word• Compete with web search engines – the
Google Test
4
Human Cataloguing of the Web
• Pros: High quality, domain knowledge selection, subject-specialised, cataloguing done to well-known and developed standards
• Cons: Expensive, slow, descriptions need to be reviewed regularly to keep them relevant
5
Software running web crawls
• Pros: vastly comprehensive (Con: too much), can be very up-to-date
• Cons: cannot distinguish “this page sucks” from “this page rocks”, indiscriminate, subject to spamming, very general (but…)
6
Combining Web Crawling and High Quality DescriptionA solution• Seed the web crawl from high quality
records• Crawl to other (presumably) good
quality pages• Track the provenance of the crawled
pages• Provenance can be used for querying
and result ranking
7
Web Search Environments (WSE) Project• Research by ILRT and later
Resource Discovery Network (RDN)• RDN funds UK SGs (ILRT also had
DutchESS)
8
WSE Technologies
• Simple Dublin Core (DC) records extracted from SGs
• OAI protocol used to collect these records in one place (not required)
• Combine Web Crawler• RDF framework to connect the
resource descriptions together
9
Simple DC Records
Really simple:• Title• Description• Identifier (URI of resource)• Source (URI of record)
10
Information model 1
• DC records describe all the resources
• Web crawler reads these and returns crawled web pages
• These generate a new web crawled resource
11
Information model 2
• Link back to original record(s), plus web page properties
• RDF model lets these be connected via page, record URIs
• Giving one large RDF graph of the total information
12
WSE graph
13
Novel Outcomes?
It is obvious that:• Metadata gathering is not new
(Harvest)• Web crawling is not new (Lycos)• Cataloguing is not new (1000s of
years)So what is new?
14
WSE – Areas Not Focused
I digress…• Gathering data together – not crucial,
Combine is a distributed harvester• Full text indexing – not optimised• Web crawling algorithm – the routes
through the web were not selected in a sophisticated way
15
WSE – General Benefits
• Connecting separate systems (one less place needed to go)
• RDF graph allows more data mixing (not fragile)
• Leverages existing systems (Combine, Zebra), standards (RDF, DC)
16
WSE – Novel Searching
• “game theory napster” – zero hits• Cross-subject searching in one
system – “gmo”• Can navigate resulting provenance
17
WSE – Gains
• Web crawling gains from high quality human description
• SGs gain from increase in relevant pages
• Fresher content than human-catalogued resource
• More focused than a general search engine
18
WSE as a new tool
• For subject experts• Which includes cataloguers• Gives fast, relevant search
(no formal precision, recall analysis)
19
WSE – new areas
• Cross-subject searching possible in subjects not yet catalogued, or that fall between SGs
• Searching emerging topics is possible ahead of additions to catalogue standards
• Helps indicate where new SGs, thesauri are needed
20
WSE - deploying
• ILRT WSE• RDN WSE• RDN – investigating for the main
search system
21
WSE for SGs
Individual SGs – enhancing subject-specific searches:
• Deep / full web crawling of high quality sites
• Granularity of cataloguing and costIt is better for humans to describe entire sites (or large parts) and let the software do the detailed work of individual pages
22
Future
• Improve and target the crawling• Use the SG information with result
ranking• Add other relevant data to the
graph such as RSS news• A Semantic Web application
23
Questions?
• Thank You• Slides:
http://ilrt.org/people/cmdjb/talks/tnc/2002/
• Project:http://wse.search.ac.uk/
24
References
• Combine Web Crawler: http://www.lub.lu.se/combine/
• Dublin Core: http://dublincore.org/ • ILRT: http://ilrt.org/ • RDF: http://www.w3.org/RDF/ • Semantic Web: http://www.w3.org/2001/
sw/
Recommended