28 Aug
2016
28 Aug
'16
10:42 p.m.
There's nothing on the webserver except a test site I use. Just trying to keep out the ones that ignore robots.txt
If its just a test server, then I'd be tempted to use HTTP AUTH at the top level. Most robots will be blocked by that, and you can use iptables to block the ones that try to guess your password, perhaps with fail2ban.
--
Kahlil (Kal) Hodgson GPG: C9A02289
Chief Technology Officer (m) +61 (0) 4 2573 0382
Direct Pricing Exchange Pty Ltd
Suite 1415
401 Docklands Drive
Docklands VIC 3008 Australia
"All parts should go together without forcing. You must remember that
the parts you are reassembling were disassembled by you. Therefore,
if you can't get them together again, there must be a reason. By all
means, do not use a hammer." -- IBM maintenance manual, 1925