Thursday, December 1, 2016

Do You know the Robots META tag

Proper Robot Tag Usage

There is a special HTML meta tag to tell robots not to index the content of a page, and/or not scan it for links to follow.

Example Usage: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

There are two important considerations when using the robots Meta Tag.
  1. robots can ignore your Meta Tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  2. the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (maybe on some other site), and so still arrives at your undesired page.

How to write a Robots Meta Tag

Where to put it

Like any Meta Tag it should be placed in the HEAD section of an HTML page. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.

What to put into it

The "NAME" attribute must be "ROBOTS".  Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW".  Multiple comma-separated values are allowed, but only some combinations should be used.  If there is no robots Meta Tag, the default assumed value is "INDEX,FOLLOW", so there's no need to spell that out.  So That leaves the following values:

  • NOINDEX, FOLLOW
  • INDEX, NOFOLLOW
  • NOINDEX, NOFOLLOW
So double check what your hoping to accomplish and add your robot Meta Tag.