#For more Robots.txt information, please see http://www.robotstxt.org. # Allowed Robots #User-agent: * #Allow: / User-agent: Slurp Crawl-delay: 40 #Allow: / User-agent: twiceler Crawl-delay: 50 #Allow: / User-agent: msnbot Crawl-delay: 50 #Allow: / # Blocked Robots User-agent: ShopWiki Disallow: / User-agent: gigabot Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: YodaoBot Disallow: / User-agent: baiduspider Disallow: /