An Attempt to Ban Bad Bots Crawling My Sites
php
dev.to
I don't really like bad bots, and by that I mean crawlers that don't care about robots.txt. The reason is simple: I don't want my data fed into obscure systems, and also just by principle, if we give you rules, follow them. Credit where it's due: the idea came from Caolan's website. The idea is simple: make the bad bots click a link they aren't supposed to, then ban them. To do that, I added a robots.txt at the root of my site, explicitly disallowing robots from a specific page (I went with /r