A handler for robots.txt, as described in http://www.robotstxt.org/wc/norobots-rfc.html.