Tutorial on setting up a robots.txt to exclude search engine robots/spiders as part of the Robots Exclusion Standard.
www.thesitewizard.com/archive/robotstxt.shtml - Similar

robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.59 2010/01/29 15:52:50 ted Exp $ # # For use by search.w3.org User-agent: W3C-gsa Disallow: ...
www.w3.org/robots.txt - Similar

If you care about validation, this robots.txt validator is a tester that will check your robots.txt file searching for syntax errors.
tool.motoricerca.info/robots-checker.phtml (Italy) - Similar

Crawlers and other Web robots are the plague of today's InterWebs. Some bots like search engine crawlers behave (IOW respec...
sebastians-pamphlets.com/smart-robots-txt/

The robots.txt is a much misunderstood file. First, it only works with those bots that obey it (such as Google) so its not really a “privacy” option. ...
www.mattcutts.com/blog/robots-txt-remove-url/ - Similar

An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion ...
radar.oreilly.com/2009/11/robotstxt-and-the-gov-tld.html

See <URL:http://www.robotstxt.org/wc/exclusion.html#robotstxt> # # Comments to the webmaster should be posted at <URL:http://www.ibm.com/contact> # # Format ...
www.ibm.com/robots.txt - Similar

robots.txt is a useful file which sits in your web site's root and controls how search engines index your pages. One of the most useful declarations is ...
www.sitepoint.com/.../why-pages-disallowed-in-robots-txt-still-appear-in- google/

While others are busy carving pumpkins or looking for the perfect mask, Google put a special Halloween egg into their robots.txt: ...
blogoscoped.com/archive/2009-10-31-n79.html

A robots.txt file restricts access to your site by search engine robots that ... To use a robots.txt file, you'll need to have root access to your server. ...
www.google.com/intl/ru/remove.html