Do You know the Robots META tag

Proper Robot Tag Usage

There is a special HTML meta tag to tell robots not to index the content of a page, and/or not scan it for links to follow.

Example Usage: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

 There are two important considerations when using the robots Meta Tag.
  1. robots can ignore your Meta Tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  2. the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (maybe on some other site), and so still arrives at your undesired page.

How to write a Robots Meta Tag

Where to put it

Like any Meta Tag it should be placed in the HEAD section of an HTML page. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.

What to put into it

The "NAME" attribute must be "ROBOTS".  Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW".  Multiple comma-separated values are allowed, but only some combinations should be used.  If there is no robots Meta Tag, the default assumed value is "INDEX,FOLLOW", so there's no need to spell that out.  So That leaves the following values:

  • NOINDEX, FOLLOW
  • INDEX, NOFOLLOW
  • NOINDEX, NOFOLLOW
So double check what your hoping to accomplish and add your robot Meta Tag.

Comments

Post a Comment

Popular posts from this blog

Do I need to build a Sitemap for my Website?

What is a sitemap? Human readable sitemaps are static HTML files that outline the first and second level structure of a Web site. The original purpose of a site map was to enable users to easily find items on the Web site.  But the purpose grew over time to became useful as a shortcut method to help search engines find and index all the parts of a site. Now, we have an XML sitemap, which effectively provides an easy-to-read link dump for the spiders to index. Some Web browsers can display an XML sitemap for users to read as well, you should offer both kinds of site maps (HTML and XML) if you want to be sure to cover both the search engines and your users. What is a Sitemap used for? Building a website can be a long tedious process, which can be more complicated by the huge amount of information that should be organized before including in your website. Some designers will start by creating  wireframes and mockups.  But for the rest of us it a lot easier to just build a sitemap
Read more

Do I need a robots.txt File?

The robots.txt file The robots.txt file to give instructions about their site to web robots; this is called  The Robots Exclusion Protocol .   Here why it gets used; a robot wants to visits a Web site URL, say http://www.myorg.com/index /.  But, before it does so, it firsts checks for http://www.myorg.com/robots.txt , and find that it contains the following: User-agent: googleboot Allow: / Allow: /images/ Allow: /css/ Allow: /js/ Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /includes/ Disallow: /processors/ Disallow: /skins/ User-agent: bingbot Allow: / Allow: /images/ Allow: /css/ Allow: /js/ Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /includes/ Disallow: /processors/ Disallow: /skins/ User-agent: * Allow: / Allow: /images/ Allow: /css/ Allow: /js/ Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /includes/ Disallow: /processors/ Disallow: /skins/ sitemap: http://www.myorg.com/sitemap.xml At MyO
Read more