Google official blog has answered Danny Sullivan of Search Engine Land in this topic,

Google bots by default crawl your site and index the contents. In robotstxt.org the user guide explains  about the usage of meta tags. In order to avoid crawling due to various reasons the Meta tag can be of this way,

<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

The value for content can be either NOINDEX or NOFOLLOW or both. When more than one value is given for content the value must be comma separated. The value NONE is same as the above value.

<META NAME=”ROBOTS” CONTENT=”NONE”>

The value INDEX or FOLLOW is unnecessary as the bots does the same job in default. Other valid values for CONTENT from the blog was,

• NOINDEX - prevents the page from being included in the index.
• NOFOLLOW - prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
• NOARCHIVE - prevents a cached copy of this page from being available in the search results.
• NOSNIPPET - prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
• NOODP - blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
• NONE - equivalent to “NOINDEX, NOFOLLOW”.

The detailed explanation is present in Google official blog.

Mark

SEO Tags: Meta Tags, Google Bots, Crawlers, Google Index

This entry was posted on Tuesday, March 6th, 2007 at 9:53 am and is filed under Google, Search Engine. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.