A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site ...
www.google.com/support/webmasters/bin/answer.py?answer... - Similar

Tutorial on setting up a robots.txt to exclude search engine robots/spiders as part of the Robots Exclusion Standard.
www.thesitewizard.com/archive/robotstxt.shtml - Similar

robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.58 2009/10/30 22:50:57 gerald Exp $ # # For use by search.w3.org User-agent: W3C-gsa Disallow: ...
www.w3.org/robots.txt - Similar

Slurp obeys the first entry in the robots.txt file with a User-agent containing "Slurp. ... If a page has robots.txt standards disallowing it to be crawled, ...
help.yahoo.com/l/us/yahoo/search/indexing/slurp-02.html - Similar

The robots.txt is a much misunderstood file. First, it only works with those bots that obey it (such as Google) so its not really a “privacy” option. ...
www.mattcutts.com/blog/robots-txt-remove-url/ - Similar

See <URL:http://www.robotstxt.org/wc/exclusion.html#robotstxt> # # Comments to the webmaster should be posted at <URL:http://www.ibm.com/contact> # # Format ...
www.ibm.com/robots.txt - Similar

The robots.txt file is the mechanism almost all search engines use to allow website administrators to tell the bots what they would like indexed. ...
drupal.org/node/22265 - Similar

These robots.txt tips will help you setup your robots.txt file to help with SEO and exclude unwanted bots on Wordpress and Joomla.
beginlinux.com/blog/2010/01/robots-txt-tips-for-deailing-with-bots/

If you don't like it, just add a robots.txt file that tells Google not to index ..... hack Murdoch's websites, but quietly change only their robots.txt to ...
boingboing.net/2009/11/08/rupert-murdoch-vows.html