# robots.txt User-agent: SemblrCrawler/Nutch-1.0 Disallow: User-agent: * Disallow: *