Yauhen KhutarniukHead of SEO at Link-Assistant.Com

@LinkAssistant

Facebook.com/linkassistant

Taming the Spiders:Optimize Your Crawl Budget to Boost Indexation and Rankings

What’s coming?• Understanding crawl budget• Measuring crawl budget• How-to: make the most of your crawl budget

Crawl Budget?

How Does Google Calculate Crawl Budget?Google Patent US8666964 B1 ‘Managing items in crawl schedule’https://www.google.ch/patents/US8666964

Page Importance• Page importance• Page authority• Page Rank• You name it

Page Rank

"The number of pages that we crawl is roughly proportional to your PageRank".

Frequency of Content Change

First Crawl Second Crawl

Significant Change

Determining your crawl budget

Server Logs

/var/log/httpd/access_log

/var/log/apache2/access.log

/var/log/httpd-access.log

How to: make the most of your crawl budget

WebSite Auditor

Download short link: http://seopwr.st/wsadownload

Make sure important pages are crawlable, and content that won't provide value if

found in search is blocked.

1

1

Avoid Long Redirect Chains

2

2

Manage URL parameters

3

3

Find and Fix Broken Links

4

4

Make Use of RSS

5

Keep Sitemap Clean and Up-to-Date

6

6

7

Website Structure and Internal Linking