Detecting Sequences and Cycles of Web Pages

Narayan L. Bhamidipati and

Sankar K. Pal

Indian Statistical InstituteKolkata

Contents

• Introduction• Objective• Significance• Procedure• Experiments• Future directions

The Web: A Directed Graph

• (V, A)• Vertices Web pages

• V = {v1, v2, …, vN}

• Arcs Hyperlinks• A = {eij : vj vi}

• Path: p1.p2. … .pn with arcs from pi to pi+1

• Cycle: A Path with pn = p1

Sequences of Web Pages

• Paths consisting of adjacent web pages• Order sensitive• A surfer may follow one such sequence

when browsing pages

Cycles of Web Pages• http://www.stanford.edu/• http://www.stanford.edu/home/atoz/letterw.html• http://www.stanford.edu/group/wellspring/• http://www.stanford.edu/group/wellspring/yahoo_spotlight.html• http://www.yahoo.com/• http://dir.yahoo.com/Education/• http://dir.yahoo.com/Education/Higher_Education/• http://dir.yahoo.com/Education/Higher_Education/Colleges_and_Universities/• http://dir.yahoo.com/Education/Higher_Education/

Colleges_and_Universities/United_States/• http://www.stanford.edu

What are we looking for ?

• A particular kind of sequences and cycles• Regular• Consisting of similar units• Units having similar relationship• Reasonably sized

Why are these Sequences and Cycles Interesting ?

• Individual units form a single object• These were intended to be together• They collectively include the complete

information• Despite being part of a collection,

individuality is maintained

Significance of Detecting Such Sequences and Cycles

• Compression• Merge groups of pages• Fewer pages fewer links

• Pre-fetching• Know where the surfer wants to be next• Fetch the page(s) before being requested• Saves time• Errors: pre-fetching wrong pages

Significance of Detecting Such Sequences and Cycles (Contd.)

• Fair comparison• Comparison independent of how content is

presented• Content split into multiple pages should be

treated equivalent to the same in a single page• Better retrieval

• Retrieval independent of the presentation• Output a set of pages instead of a single one as

a match

Fair Comparison

Fair Comparison

Fair Comparison

Improved Retrieval

• Retrieve only portions of interest• Instead of, whole (huge) documents• Avoid rewarding more content

How to Detect Sequences and Cycles of Web Pages ?

• Find navigational links• Find consecutive pages

• Define what the elements of the sequence would satisfy

• Identify subsequences (or units)• Concatenate

• Check for cycles

Finding Navigational Links: Background

• The purpose of a link may be• Navigation• Reference• Advertisement

• Links between pages on the same server are treated as navigational

• Have also been treated as noise

Finding Navigational Links: Our Method

• Avoid treating links on the same server as navigational links

• Appear mostly either at the top or at the bottom

• Navigational links are generally huddled together

• Fewer text and images around such links

Advantages and Limitations

• Simple and fast• Navigational links across servers are also

identified

• Heuristics need not always work – fall back on sophisticated methods

Units of the Sequences

• ABC is a unit if C is “related” to B in the same way as B is “related” to A

• “related” is defined in terms of how they are linked

• Relation is stored as “position” of the link• Several ways of defining “position”

Combining the units into sequences

• DEF• BCD• ABC• CDE

• ABCDEF

Cycle detection

• Existing cycle detection algorithms• Cycle detection in number theory• Special case of cycle detection in graph

theory• Stack based algorithm

Improvements and Speedups

• Believe the “rel” information provided by the (author of the) pages

• Use keywords like “next” and “previous” to perceive the relationships

• Utilize the information of the naming convention

Experimental Results

• Data• Toy data: python tutorial in HTML• Tutorial split into several chapters and sections• Several cycles

• Mutilated data• Certain pages deleted (missing links)

• 100% detection in all cases

Other experiments planned

• Real test: unorganized web pages• Difficulties:

• Finding navigational links• Noise (advertisements, etc)• Dynamically generated

• Will the relationships hold ?

Leads us to …

• Concatenate detected sequences for analysis• Modify retrieval mechanism• Return sets of pages as results• Improve mirror/duplicate detection

Future Work

• Consider other relations• Unifying framework ?• Improve identification of navigational links