如何阻止坏机器人，蜘蛛，履带和收割机

我厌倦了这个糟糕的机器人，蜘蛛，爬行器和收割机。 我已经configuration我的服务器阻止连接IP 5分钟和maxretry 250使用fail2ban。 但是仍然有一部分人在250分钟内无法拦截，因为他们在5分钟内没有访问250个以上。

这是我的jail.localconfiguration：

[http-get-dos] enabled = true filter = http-get-dos logpath = /var/log/ispconfig/httpd/*/access.log maxretry = 250 findtime = 300 #ban for 10 hours bantime = 36000 action = iptables-multiport[name=HTTP, port="http,https", protocol=tcp] cloudflare-blacklist sendmail-whois[name=HTTP, [email protected]] 

这里是http-get-dos.conffilter文件：

 [Definition] failregex = ^<HOST> -.*"(GET|POST) ignoreregex = 

可以阻止这个爬虫的教程的大部分是使用Apache。 但是因为我使用nginx，所以我不能使用它们。 这是我find的一个教程 。

• 脚本或某人观看从我的服务器（HTTP）下载文件的人
• Munin限制COUNTER和DERIVE数据
• 我如何知道应用程序使用的内存
• 如何在ubuntu屏幕上的进程停止时得到电子邮件通知？
• 我如何监视和绘制这个WMI / WQL数据？

这里是这个机器人的例子日志：

 220.225.127.41 - - [24/Jul/2013:00:00:19 +0800] "GET /php?page=9 HTTP/1.1" 200 10897 "http://www.mysite.com/php?page=8" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:22 +0800] "GET /sites/default/files/download/jkev/jkev_search.zip HTTP/1.1" 200 35199 "http://www.mysite.com/sites/default/files/download/jkev/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:00:00:24 +0800] "GET /sites/default/files/styles/thumbnail/public/images/kalola/sk_3.jpg?itok=-pXuOEq2 HTTP/1.1" 200 3958 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:24 +0800] "GET /sites/default/files/styles/thumbnail/public/images/kalola/sk_1.jpg?itok=ug6jsTPP HTTP/1.1" 200 3958 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:24 +0800] "GET /sites/default/files/styles/thumbnail/public/images/kalola/sk_2.jpg?itok=ZPOMnJeK HTTP/1.1" 200 3958 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:26 +0800] "GET /sites/default/files/styles/thumbnail/public/images/argie/currency.jpg?itok=hodqOr4_ HTTP/1.1" 200 7976 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:26 +0800] "GET /sites/default/files/styles/thumbnail/public/images/localhost27/untitled.jpg?itok=uVeczDjI HTTP/1.1" 200 3136 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:26 +0800] "GET /sites/default/files/styles/thumbnail/public/images/Oelasor/screenshot_11.jpg?itok=uu3d0GpX HTTP/1.1" 200 6674 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:27 +0800] "GET /sites/default/files/styles/thumbnail/public/images/localhost27/member.jpg?itok=inA9ULoC HTTP/1.1" 200 4500 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:28 +0800] "GET /php/4852/shopping-cart-checkout-using-codeigniter.html HTTP/1.1" 200 11414 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:29 +0800] "GET /sites/default/files/styles/medium/public/images/admin/codeigniter_shopping_cart.jpg?itok=QO0YV6JP HTTP/1.1" 200 22534 "http://www.mysite.com/php/4852/shopping-cart-checkout-using-codeigniter.html" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:32 +0800] "GET /php/4846/simple-ajax-example-php.html HTTP/1.1" 200 10174 "http://www.mysite.com/php?page=9" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" 220.225.127.41 - - [24/Jul/2013:00:00:34 +0800] "GET /sites/default/files/download/teejaygenius/e_library.zip HTTP/1.1" 206 3655400 "http://www.mysite.com/sites/default/files/download/teejaygenius/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:00:00:36 +0800] "GET /sites/default/files/download/Chritian/bus.zip HTTP/1.1" 206 4462491 "http://www.mysite.com/sites/default/files/download/Chritian/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:00:00:37 +0800] "GET /sites/default/files/styles/medium/public/images/kalola/sk_2.jpg?itok=1N0a__bq HTTP/1.1" 200 9693 "http://www.mysite.com/php/4846/simple-ajax-example-php.html" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36"e220.225.127.41 - - [24/Jul/2013:03:03:13 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 1555432 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:20 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18541381 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:29 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 6186320 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:31 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 13495467 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:34 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 17908605 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:51 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 10082448 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:03:57 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 8639709 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:03 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 12150765 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:04 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 17972316 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:09 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18453052 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:23 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 777716 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:40 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 8033075 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:45 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 12935983 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:49 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 8262600 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:49 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 11598966 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:49 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 11249310 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:04:57 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 5969210 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:02 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 12978641 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:03 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 13390784 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:07 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 6124786 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:15 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 9962834 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:19 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 12021359 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:27 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 8432875 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:44 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18371964 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:46 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 19867749 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:05:50 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18164900 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:00 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 17839100 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:01 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18329973 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:11 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18651902 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:31 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 9858200 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:34 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 12914955 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:36 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 13315966 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:38 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 12804285 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:41 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 6043976 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:42 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 11900897 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:06:52 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 8293782 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:07:06 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 11582412 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:07:24 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18667357 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:07:27 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 7977266 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:07:35 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 11190040 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:07:36 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18555860 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:09 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 5932064 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:10 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 12730175 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:13 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 13208853 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:16 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 8178860 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:22 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 5896753 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:25 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 8183834 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:26 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 12671818 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:30 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18581925 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:36 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18224268 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:37 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 11761743 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:51 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 11412627 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:08:59 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 18600749 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:09:01 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 11129155 "http://www.mysite.com/sites/default/files/download/mindgamez/" "FDM 3.x" 220.225.127.41 - - [24/Jul/2013:03:09:14 +0800] "GET /sites/default/files/download/argie/tameraplazainn.zip HTTP/1.1" 206 7836467 "http://www.mysite.com/sites/default/files/download/argie/" "FDM 3.x" 

这是按小时访问的频率：

 # grep "220.225.127.41" /var/log/ispconfig/httpd/*/access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c 545 00:00 524 01:00 404 02:00 491 03:00 396 04:00 183 05:00 

以下是分钟访问的频率（午夜12点左右）：

 # grep "220.225.127.41 - - \[24/Jul/2013:00" /var/log/ispconfig/httpd/*/access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}' 33 00:00 14 00:01 12 00:03 26 00:05 15 00:10 18 00:11 22 00:13 15 00:14 14 00:15 15 00:18 21 00:19 17 00:20 15 00:23 14 00:24 17 00:25 27 00:29 15 00:30 18 00:32 14 00:52 

以下是分钟访问的频率（大约凌晨1点）：

 # grep "220.225.127.41 - - \[24/Jul/2013:01" /var/log/ispconfig/httpd/*/access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}' 16 01:01 16 01:02 12 01:05 16 01:06 14 01:10 14 01:11 14 01:12 13 01:14 22 01:16 18 01:17 13 01:21 21 01:22 14 01:26 20 01:37 30 01:38 13 01:45 11 01:50 17 01:51 11 01:53 

有没有办法阻止这个使用IPTables什么的？

如果我降低，恐怕一些合法的stream量也会被禁止。

访问率非常低。 我不能将最大值设置为50甚至70，这也会阻止合法的stream量。

那我怎么能阻止这个呢？ 他们消耗太多的带宽。 在我的正常带宽为每天59.31 GB之前，现在达到了136.74 GB。

 -A INPUT -j ACCEPT -p tcp --dport 80 -s xxx.yyy.0.0/16 --syn -m connlimit ! --connlimit-above 20 -A INPUT -j ACCEPT -p tcp --dport 80 --syn -m connlimit ! --connlimit-above 5 --connlimit-mask 24