Upload
michi
View
43
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Antonio Gulli. AGENDA. Overview of Spidering Technology … hey, how can I get that page? Overview of a Web Graph … tell me something about the arena Google Overview. - PowerPoint PPT Presentation
Citation preview
Antonio Gulli
AGENDA Overview of Spidering
Technology … hey, how can I get that page?
Overview of a Web Graph… tell me something about the arena
Google OverviewMining the WebDiscovering Knowledge from Hypertext DataSoumen Chakrabarti (CAP.2)Morgan-Kaufmann Publishers352 pages, cloth/hard-boundISBN 1-55860-754-4
Spidering 24h, 7days “walking” over a Graph, getting
data
What about the Graph? Recall the sample of 150 sites (last lesson) Direct graph G = (N, E) N changes (insert, delete) ~ 4-6 * 109 nodes E changes (insert, delete) ~ 10 links for node Size 10*4*109 = 4*1010 non zero entries in adj
matrix EX: suppose a 64 bit hash for URL, how much
space for storing the adj matrix?
A Picture of the Web Graph
[DelCorso, Gulli, Romani .. WAW04]
i
j Q: sparse or not sparse?
21 milions of pages, 150millions of links
A Picture of the Web Graph
[BRODER, www9]
A Picture of the Web Graph
Stanford
Berkeley
[Hawelivala, www12]
Q: what kind of sorting is this?
A Picture of the Web Graph
[Ravaghan, www9] READ IT!!!!!
The Web’s Characteristics Size
Over a billion pages available 5-10K per page => tens of terabytes Size doubles every 2 years
Change 23% change daily Half life time of about 10 days Bowtie structure
Search Engine Structure
Crawl Control
CrawlersRanking
Indexer
Page Repository
Query Engine
Collection Analysis
Text Structure Utility
Queries Results
Indexes
Bubble ???
Link Extractor:while(<ci sono pagine da cui estrarre i link>){ <prendi una pagina p dal page repository> <estrai i link contenuti nel tag a href> <estrai i link contenuti in javascript> <estrai ….. <estrai i link contenuti nei frameset> <inserisci i link estratti nella priority que, ciascuna con una priorità dipendente dalla politica scelta e: 1) compatibilmente ai filtri applicati 2) applicando le operazioni di normalizzazione> <marca p come pagina da cui abbiamo estratto i link>}
Downloaders:while(<ci sono url assegnate dai crawler manager>){ <estrai le url dalla coda di assegnamento> <scarica le pagine pi associate alla url dalla rete> <invia le pi al page repository>}
Crawler Manager:<estrai un bunch di url dalla “priority que” in ordine>while(<ci sono url assegnate dai crawler manager>){ <estrai le URL ed assegnale ad S> foreach u S {
if ( (u “Already Seen Page” ) || ( u “Already Seen Page” && (<sul Web server la pagina è più recente> )
&& ( <u è un url accettata dal robot.txt del sito>) ) { <risolvi u rispetto al DNS> <invia u ai downloaders, in coda>
}}
Crawler “cycle of life”
Architecture of Incremental Crawler
Strutture Dati
Moduli Software
SPIDERSINDEXERS
ParallelDownloaders
DNS Revolvers
DNS Cache
Parallel Crawler Managers
AlreadySeen Pages
Robot.txtCache
Parallel Link Extractors
PriorityQue
Distributed Page Repository
Parsers
INTERNET
Page Analysis
Indexer
LEGENDA
……
…
[Gulli, 98]
Crawling Issues How to crawl?
Quality: “Best” pages first Efficiency: Avoid duplication (or near duplication) Etiquette: Robots.txt, Server load concerns (Minimize load)-
How much to crawl? How much to index? Coverage: How big is the Web? How much do we cover? Relative Coverage: How much do competitors have?
How often to crawl? Freshness: How much has changed? How much has really changed? (why is this a different
question?)
How to parallelize the process
Page selection Crawler method for choosing page to download Given a page P, define how “good” that page is. Several metric types:
Interest driven Popularity driven (PageRank, full vs partial) BFS, DFS, Random Combined Random Walk
Potential quality measures: Final Indegree Final Pagerank
BFS “…breadth-first search order discovers the highest
quality pages during the early stages of the crawl BFS” 328 milioni di URL nel testbed
[Najork 01]
Q: how this is related to SCC, Power Laws. domains hierarchy in a Web Graph?
See more when we will do PageRank
Perc. Overlap with best x% by indegree
x% crawled by O(u) x% crawled by O(u)
Stanford Web Base (179K, 1998)[Cho98]
Perc. Overlap with best x% by pagerank
BFS & Spam (Worst case scenario)
BFS depth = 2
Normal avg outdegree = 10
100 URLs on the queue including a spam page.
Assume the spammer is able to generate dynamic pages with 1000 outlinks
StartPage
StartPage
BFS depth = 32000 URLs on the queue50% belong to the spammer
BFS depth = 41.01 million URLs on the queue99% belong to the spammer
Can you trust words on the page?
Examples from July 2002
auctions.hitsoffice.com/
www.ebay.com/Pornographic Content
A few spam technologies Cloaking
Serve fake content to search engine robot DNS cloaking: Switch IP address.
Impersonate Doorway pages
Pages optimized for a single keyword that re-direct to the real target page
Keyword Spam Misleading meta-keywords, excessive
repetition of a term, fake “anchor text” Hidden text with colors, CSS tricks, etc.
Link spamming Mutual admiration societies, hidden links,
awards Domain flooding: numerous domains that
point or re-direct to a target page Robots
Fake click stream Fake query stream Millions of submissions via Add-Url
Is this a SearchEngine spider?
Y
N
SPAM
RealDoc
Cloaking
Meta-Keywords = “… London hotels, hotel, holiday inn, hilton, discount, booking, reservation, sex, mp3, britney spears, viagra, …”
Parallel Crawlers• Web is too big to be crawled by a single
crawler, work should be divided• Independent assignment
• Each crawler starts with its own set of URLs• Follows links without consulting other
crawlers• Reduces communication overhead• Some overlap is unavoidable
Parallel Crawlers• Dynamic assignment
• Central coordinator divides web into partitions• Crawlers crawl their assigned partition• Links to other URLs are given to Central coordinator
• Static assignment• Web is partitioned and divided to each crawler• Crawler only crawls its part of the web
URL-Seen Problem Need to check if file has been parsed or
downloaded before - after 20 million pages, we have “seen” over
100 million URLs - each URL is 50 to 75 bytes on average Options: compress URLs in main memory, or
use disk - Bloom Filter (Archive) [we will discuss this
later] - disk access with caching (Mercator,
Altavista)
Virtual Documents P = “whitehouse.org”, not yet reached {P1….Pr} reached, {P1….Pr} points to P Insert into the index the anchors context
…George Bush, President of U.S. lives at <a href=http://www.whitehouse.org> WhiteHouse</a>
Washington
Pagina Web e Documento Virtuale
bushWhite House
Focused Crawling Focused Crawler: selectively seeks out
pages that are relevant to a pre-defined set of topics.
- Topics specified by using exemplary documents (not keywords)
- Crawl most relevant links- Ignore irrelevant parts.- Leads to significant savings in hardware
and network resources.
Focused Crawling
niHjHEEP
HHEEH
jj
iii ,....,1
)Pr()|Pr()()Pr()|Pr()|Pr(
Pr[documento rilevante | il termine t è presente]Pr[documento irrilevante | il termine t è presente]Pr[termine t sia presente | il doc sia rilevante] Pr[termine t sia presente | il doc sia irrilevante]
An example of crawler Polybot crawl of 120 million pages over 19 days 161 million HTTP request 16 million robots.txt requests 138 million successful non-robots requests 17 million HTTP errors (401, 403, 404 etc) 121 million pages retrieved slow during day, fast at night peak about 300 pages/s over T3 many downtimes due to attacks, crashes, revisions http://cis.poly.edu/polybot/
[Suel 02]
Examples: Open Source
Nutch, also used by Overture http://www.nutch.org
Hentrix, used by Archive.org http://archive-crawler.sourceforge.net/index.html
Where we are? Spidering Technologies Web Graph (a glimpse)
Now, some funny math on two crawling issues
1) Hash for robust load balance 2) Mirror Detection
Consistent Hashing A mathematical tool
for: Spidering Web Cache P2P Routers Load Balance Distributed FS
Item and servers ← ID ( hash function of m bits)
Node identifier mapped on a 2^m ring
Item K assigned to first server with ID ≥ k
What if a downloader goes down?
What if a new downloader appear?
Duplicate/Near-Duplicate Detection
Duplication: Exact match with fingerprints Near-Duplication: Approximate match
Overview Compute syntactic similarity with an edit-
distance measure Use similarity threshold to detect near-
duplicates E.g., Similarity > 80% => Documents are “near
duplicates” Not transitive though sometimes used transitively
Computing Near Similarity Features:
Segments of a document (natural or artificial breakpoints) [Brin95]
Shingles (Word N-Grams) [Brin95, Brod98] “a rose is a rose is a rose” => a_rose_is_a rose_is_a_rose is_a_rose_is
Similarity Measure TFIDF [Shiv95] Set intersection [Brod98] (Specifically, Size_of_Intersection / Size_of_Union )
Shingles + Set Intersection Computing exact set intersection of shingles between all pairs of documents is expensive and infeasible Approximate using a cleverly chosen subset of
shingles from each (a sketch)
Shingles + Set Intersection Estimate size_of_intersection / size_of_union based on a short sketch ( [Brod97, Brod98] ) Create a “sketch vector” (e.g., of size 200) for
each document Documents which share more than t (say 80%)
corresponding vector elements are similar For doc D, sketch[ i ] is computed as follows:
Let f map all shingles in the universe to 0..2m (e.g., f = fingerprinting)
Let i be a specific random permutation on 0..2m
Pick sketch[i] := MIN i ( f(s) ) over all shingles s in D
Computing Sketch[i] for Doc1Document 1
264
264
264
264
Start with 64 bit shingles
Permute on the number linewith i
Pick the min value
Test if Doc1.Sketch[i] = Doc2.Sketch[i]
Document 1 Document 2
264
264
264
264
264
264
264
264
Are these equal?
Test for 200 random permutations: , ,… 200
A B
However…Document 1 Document 2
264
264
264
264
264
264
264
264
A = B iff the shingle with the MIN value in the union of Doc1 and Doc2 is common to both (I.e., lies in the intersection)
This happens with probability: Size_of_intersection / Size_of_union
BA
Mirror Detection Mirroring is systematic replication of web
pages across hosts. Single largest cause of duplication on the
web Host1/ and Host2/ are mirrors iff
For all (or most) paths p such that when http://Host1/ / p exists http://Host2/ / p exists as wellwith identical (or near identical) content,
and vice versa.
Mirror Detection example http://www.elsevier.com/ and
http://www.elsevier.nl/ Structural Classification of Proteins
http://scop.mrc-lmb.cam.ac.uk/scop http://scop.berkeley.edu/ http://scop.wehi.edu.au/scop http://pdb.weizmann.ac.il/scop http://scop.protres.ru/
Motivation Why detect mirrors?
Smart crawling Fetch from the fastest or freshest server Avoid duplication
Better connectivity analysis Combine inlinks Avoid double counting outlinks
Redundancy in result listings “If that fails you can try: <mirror>/samepath”
Proxy caching
Maintain clusters of subgraphs Initialize clusters of trivial subgraphs
Group near-duplicate single documents into a cluster Subsequent passes
Merge clusters of the same cardinality and corresponding linkage
Avoid decreasing cluster cardinality To detect mirrors we need:
Adequate path overlap Contents of corresponding pages within a small time range
Bottom Up Mirror Detection[Cho00]
Can we use URLs to find mirrors?
www.synthesis.org
a b
cd
synthesis.stanford.edu
a b
cd
www.synthesis.org/Docs/ProjAbs/synsys/synalysis.htmlwww.synthesis.org/Docs/ProjAbs/synsys/visual-semi-quant.htmlwww.synthesis.org/Docs/annual.report96.final.htmlwww.synthesis.org/Docs/cicee-berlin-paper.htmlwww.synthesis.org/Docs/myr5www.synthesis.org/Docs/myr5/cicee/bridge-gap.htmlwww.synthesis.org/Docs/myr5/cs/cs-meta.htmlwww.synthesis.org/Docs/myr5/mech/mech-intro-mechatron.htmlwww.synthesis.org/Docs/myr5/mech/mech-take-home.htmlwww.synthesis.org/Docs/myr5/synsys/experiential-learning.htmlwww.synthesis.org/Docs/myr5/synsys/mm-mech-dissec.htmlwww.synthesis.org/Docs/yr5arwww.synthesis.org/Docs/yr5ar/assesswww.synthesis.org/Docs/yr5ar/ciceewww.synthesis.org/Docs/yr5ar/cicee/bridge-gap.htmlwww.synthesis.org/Docs/yr5ar/cicee/comp-integ-analysis.html
synthesis.stanford.edu/Docs/ProjAbs/deliv/high-tech-…synthesis.stanford.edu/Docs/ProjAbs/mech/mech-enhanced…synthesis.stanford.edu/Docs/ProjAbs/mech/mech-intro-…synthesis.stanford.edu/Docs/ProjAbs/mech/mech-mm-case-…synthesis.stanford.edu/Docs/ProjAbs/synsys/quant-dev-new-…synthesis.stanford.edu/Docs/annual.report96.final.htmlsynthesis.stanford.edu/Docs/annual.report96.final_fn.htmlsynthesis.stanford.edu/Docs/myr5/assessmentsynthesis.stanford.edu/Docs/myr5/assessment/assessment-…synthesis.stanford.edu/Docs/myr5/assessment/mm-forum-kiosk-…synthesis.stanford.edu/Docs/myr5/assessment/neato-ucb.htmlsynthesis.stanford.edu/Docs/myr5/assessment/not-available.htmlsynthesis.stanford.edu/Docs/myr5/ciceesynthesis.stanford.edu/Docs/myr5/cicee/bridge-gap.htmlsynthesis.stanford.edu/Docs/myr5/cicee/cicee-main.htmlsynthesis.stanford.edu/Docs/myr5/cicee/comp-integ-analysis.html
Where we are?
Spidering Web Graph Some nice mathematical tools
Many others funny algorithmic for crawling issues…
Now, a glimpse on a Google: (thanks to Jungoo Cho, 3rd founder..)
Google: Scale
Number of pages indexed:3B in November 2002
Index refresh interval:Once per month ~ 1200 pages/sec
Number of queries per day:200M in April 2003 ~ 2000
queries/sec Runs on commodity Intel-Linux boxes
[Cho, 02]
Google: Other Statistics Average page size: 10KB Average query size: 40B Average result size: 5KB Average number of links per page: 10 Total raw HTML data size
3G x 10KB = 30 TB! Inverted index roughly the same size as
raw corpus: 30 TB for index itself With appropriate compression, 3:1
20 TB data residing in disk (and memory!!!)
Google:Data Size and Crawling Efficient crawl is very important
1 page/sec 1200 machines just for crawling
Parallelization through thread/event queue necessary
Complex crawling algorithm -- No, No! Well-optimized crawler
~ 100 pages/sec (10 ms/page) ~ 12 machines for crawling
Bandwidth consumption 1200 x 10KB x 8bit ~ 100Mbps One dedicated OC3 line (155Mbps) for
crawling ~ $400,000 per year
Google: Data Size, Query Processing Index size: 10TB 100 disks Typically less than 5 disks per machine Potentially 20-machine cluster to answer a
query If one machine goes down, the cluster goes
down Two-tier index structure can be helpful
Tier 1: Popular (high PageRank) page index Tier 2: Less popular page index Most queries can be answered by tier-1
cluster (with fewer machines)
Google: Implication of Query Load
2000 queries / sec Rule of thumb: 1 query / sec per CPU
Depends on number of disks, memory size, etc.
~ 2000 machines just to answer queries 5KB / answer page
2000 x 5KB x 8bit ~ 80 Mbps Half dedicated OC3 line (155Mbps) ~ $300,000
Google: Hardware
50,000 Intel-Linux cluster Assuming 99.9% uptime (8 hour downtime
per year) 50 machines are always down Nightmare for system administrators
Assuming 3-year hardware replacement Set up, replace and dump 50 machines
every day Heterogeneity is unavoidable
Shingles computation (for Web Clustering)
s1=a_rose_is_a_rose_in_the = w1 w2 w3 w4 w5 w6 w7 s2=rose_is_a_rose_in_the_garden = w1 w2 w3 w4 w5 w6 w7
0/1 representation (using ascii code) HP: a word is ~8 byte → length(si)=7*8 byte=448bit This represent S, a poly with coefficient in 0/1 (Z2)
Rabin fingerprint Map 448bit in a K=40bit space, with low collision prob. Generate an irreducible poly P with degree K-1 (see Galois Z2) F(S) = S mod P
No explicit random permutation better to work forcing length(si)=K*z, z in N (for instance 480bit) Induce a “random permutation” shifting of 480-448 bits (32 position)
Simple example (just 3 words, of 1 char each) S1=a_b_c_d, S2=b_c_d_e (a is 97 in ascii) S1
0/1=01100001 01100010 01100011 01100100 = 32 bit S2
0/1=01100010 01100011 01100100 01100101 = 32 bit HP K = 5 bit
S10/1=01100001 01100010 01100011 01100100 000 = 35 bit
S20/1=01100010 01100011 01100100 01100101 000 = 35 bit
Shingles computation (an example)
Shingles computation (an example)
We choose x4+x +1=10011 FOR S1
0/1
01100 mod 10011=12 mod 19 = 12 = 01100
(01100 + 00101) mod 10011 = (12 + 5) mod 19 = 17 = 10001
(10001 + 10001) mod 10011 = (17 + 17) mod 19 = 15 = 01111
(01111 + 00110) mod 10011 = (15 + 6) mod 19 = 2 = 00010
(00010 + 00110) mod 10011 = ( 2 + 6) mod 19 = 8 = 00100
(00100+11001) mod 10011 = (8 + 25) mod 19 = 14 = 01110
(01110 + 00000) mod 10001 = 01110 FINGERPRINT
FOR S20/1
We can do in O(1), EX: how????
Shingle Computation (using xor and other operations)
since coefficients are 0 or 1, can represent any such polynomial as a bit string
addition becomes XOR of these bit strings multiplication is shift & XOR modulo reduction done by repeatedly
substituting highest power with remainder of irreducible poly (also shift & XOR)
Shingles computation (final step)
Given the set of fingerprints Take 1 out of m fingerprints (instead of the
minimum) This is the set of fingerprints, for a given document D
Repeat the generation for each document D1, …, Dn
We obtain the set of tuple T ={<fingerprint, Di>, …} Sort T (external mem. sort for Syntatic Web Clustering) Linear Scan for counting adjacent tuples All those above a threshold are near-duplicate (cluster)
Shingles computation (an example)
S10/1=01100001 01100010 01100011 01100100 000
S20/1=01100010 01100011 01100100 01100101 000
S_2 = (S_1 - 01100 001 * 227) * 28 + 01100101 000 precompute 227 mod 10011= x precompute 28 mod 10011= y S_2 mod 10011 =[(S_1 mod 10011 - 01100 00 mod
10011 * x) * y + 01100101 000]mod 10011 --