# Simple robots.txt file 2 User-agent: * Disallow: /tmp/documents/ Disallow: /under_construction/ User-agent: CRAWLER Disallow: /fred