It's true pure thieves won't obey robots.txt, but less sophisticated ones will run bots that by default do obey it. So even if you only get 1 out of 10 scrappers to not scrap, that more than makes up for the few seconds it takes to put up the file. Also, there are much more benign pots that uselessly eat up bandwidth and server load that you can do without. For instance some of the link exchange bots could take thousands of pages every single day, and if you have no interest in that particular link network there is no reason to just have them stumbling around your sites every day. Those bots generally do obey robots.txt.
|