By: Dan Crow | Source: googleblog.blogspot.com

A post on official Google blog informs about Robots Exclusion Protocol. Sometime back  Google posted another post on Robots.txt file.

It imparted important details to the web publishers about how they can control indexing and accessing of sites by search engines and Google itself. The important tool for the same purpose is the robots.txt file. Robots.txt file gives powerful control to site owners on how the site is searched.

The more recent post on robots exclusion protocol provides more details and examples of mechanisms to control access and indexing of your website by Google.

This post simplifies the procedure of preventing Googlebot from following a link. “Usually when the Googlebot finds a page, it reads all the links on that page and then fetches those pages and indexes them. This is the basic process by which Googlebot "crawls" the web. This is useful as it allows Google to include all the pages on your site, as long as they are linked together.” It further says that one can add the NOFOLLOW tag to a  page which tells the Googlebot not to follow any links it finds on that page.

Read more at googleblog.blogspot.com

Spread the word: bookmark it/ readit