Google Directory - Computers > Internet > Searching > Search Engines

Directory Help
Search only in RobotsSearch the Web

Robots

Computers > Internet > Searching > Search Engines > Robots

Web Pages

Viewing in Google PageRank order View in alphabetical order

	Robotstxt.org - http://www.robotstxt.org/ Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
	ACAP - Automated Content Access Protocol - http://www.the-acap.org/ Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.
	User-Agents.org - http://www.user-agents.org/ Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.
	Bots vs Browsers - http://www.botsvsbrowsers.com This large database lists user agents in categories and distinguishes between robots and browsers.
	All About Search Indexing Robots and Spiders - http://www.searchtools.com/robots/ Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

	Search Engine Robots and Other User Agents - http://www.jafsoft.com/searchengines/webbots.html John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
	Search Engine IP Addresses - http://www.iplists.com/ Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
	List of Robot Agent Strings - http://www.pgts.com.au/pgtsj/pgtsj0208d.html A list from PGTS of Web robots with the identifying data they leave in Web site logs.
	User Agent String - http://user-agent-string.info/ Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
	HTTP User Agent Index - http://www.siteware.ch/webresources/useragents/db.html An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.

Robot IP Address - http://www.briandunning.com/seo/
Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.

Help build the largest human-edited directory on the web.

Submit a Site - Open Directory Project - Become an Editor