About the Robots <META> tag
In a nutshell
You can use a special HTML <META> tag to tell robots not to index the content of a page, and/or not scan it for links to follow.For example:
<html> <head> <title>...</title> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> </head>There are two important considerations when using the robots <META> tag:
- robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.
The details
Like the /robots.txt, the robots META tag is a de-facto standard. It originated from a "birds of a feather" meeting at a 1996 distributed indexing workshop, and was described in meeting notes.The META tag is also described in the HTML 4.01 specification, Appendix B.4.1.
The rest of this page gives an overview of how to use the robots <META> tags in your pages, with some simple recipes. To learn more see also the FAQ.
How to write a Robots Meta Tag
Where to put it
Like any <META> tag it should be placed in the HEAD section of an HTML page, as in the example above. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.What to put into it
The "NAME" attribute must be "ROBOTS".Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW". Multiple comma-separated values are allowed, but obviously only some combinations make sense. If there is no robots <META> tag, the default is "INDEX,FOLLOW", so there's no need to spell that out. That leaves:
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
SEOs use “noindex” tags to prevent a search engine robot from indexing a particular page on a site. We use “nofollow” tags to prevent the passing of page rank through a particular link to the linked-to page (aka “bot herding”, “page rank sculpting”, etc.). Nothing wrong with either of these techniques, but I often see them used in a less than subtle manner that can often do as much harm as good.
For some reason, most people use a “noindex, nofollow” tag as the default “noindex” tag when just using the “noindex” tag alone would be more appropriate. When you add the “nofollow” tag to a meta robots tag on a page, it causes all links on that page to be tagged as “nofollow”. This can be the equivalent of using a bazooka when a paint brush will do.
