• The Web Robots Pages
  • The /robots.txt checker can check your site's /robots.txt file and meta tags. The IP Lookup can help find out more about what robots are visiting you.


  • Robots exclusion standard - Wikipedia, the free encyclopedia
  • The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from ...


  • The Web Robots Pages
  • This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified below. This approach was chosen because it can be easily implemented ...


  • www.google.com
  • User-agent: * Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs. Disallow: /catalogues. Disallow: /news. Allow: /news/directory


  • Robots.txt Generator - McAnerin International Inc.
  • robots.txt generator designed by an SEO for public use. Includes tutorial.


  • Block or remove pages using a robots.txt file - Webmaster Tools Help
  • A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a ...


  • Robots.txt Information
  • Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator


  • www.whitehouse.gov
  • User-agent: * Crawl-delay: 10 . Sitemap: http://www.whitehouse.gov/feed/media/video-audio


  • Introduction to "robots.txt"
  • Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.


  • Robots.txt and Search Indexing - Search Tools Report
  • Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing certain sections of a site.