How To Block Google To Crawl and Index Your Webpages

Sponsored by: www.pagetron.com

There might be a number of reasons why you do not want Google to index and cache a particular page of your site. To block a particular page to show up in Google you have to a write a file called robots.txt and upload it to the root directory of your web server. In general to block a webpage or pages from being crawled and indexed by Google you need to follow the following steps:

1. Create a blank robots.txt file, which is just a text file.2. Write the commands to block the URL of the page that you want

to block.3. Upload the file in your web directory. The location of the file will

be under the public_html.

In your computer, open a note pad and name is as robots and save it. your computer will automatically save it as robots.txt

Now write the necessary command. for example the URL of your website is : www.pagetron.com and you have a post on it with the URL www.pagetron.com/how-to-blog and you want to block this page from getting indexed in all the search engines. So, use the following instructions:

Create a robots.txt

Add the following two lines in your robot.txt file that you have created.

User-agent: * Disallow: /how-to-blog

And then save it. at the end of the file you can add the sitemap URL of your blog, if you want.

The command user-agent : * means that you are telling every search bot to avoid the URL in the Disallow command.

The Disallow command states the URL that will be blocked. No need to put the entire URL after the Disallow command. Just write the URL that is after your website’s domain name and do not forget to add the slashes (/) before and after the URL. So, your Disallow section of the robots.txt will look like the following command:

Disallow: /how-to-blog/

Thank you!!!!

Sponsored by: www.pagetron.com