如何使用fail2ban禁止networking爬虫

我正在使用nginx，如果我是正确的，我总是被networking爬虫击中。

我试图configurationfail2ban，但IP地址不能被fail2ban检测到。

因为看起来是合法的访问者所以不被检测的理由。 以下是示例日志：

116.73.68.36 - - [19/Jul/2013:23:57:47 +0800] "GET /sites/default/files/download/rhenz23/it4cai.zip HTTP/1.1" 206 14628884 "http://www.mysite.com/php/5297/computer-aided-instruction.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 14510149 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16357796 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/tovi/online_lot_reservation_system.zip HTTP/1.1" 206 14850935 "http://www.mysite.com/php/4088/online-lot-reservation.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17217908 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/sanbunna9/online_gues_house.zip HTTP/1.1" 206 17594389 "http://www.mysite.com/php/5235/online-guest-house.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:50 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17070214 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/welmarie/online_product_reservation_system.zip HTTP/1.1" 206 15074810 "http://www.mysite.com/php/3969/online-product-reservation-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1_0.zip HTTP/1.1" 206 15232701 "http://www.mysite.com/php/4094/online-membership-and-billing-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 15555605 "http://www.mysite.com/php/4171/online-management-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16379516 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 17671134 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:57:59 +0800] "GET /sites/default/files/download/malyn30/socialnetworkingsite.zip HTTP/1.1" 206 16711108 "http://www.mysite.com/php/3971/sample-simple-social-networking-site.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:58:04 +0800] "GET /sites/default/files/download/Franziholic/franzdarylduetes.zip HTTP/1.1" 206 17718916 "http://www.mysite.com/php/5408/reyans-burger-online-ordering-system-using-php.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:58:07 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 16876180 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:58:08 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 16685045 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:58:15 +0800] "GET /sites/default/files/download/jkev/psits_voting_system.zip HTTP/1.1" 206 17465518 "http://www.mysite.com/php/5442/drag-and-drop-voting-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 116.73.68.36 - - [19/Jul/2013:23:58:32 +0800] "GET /sites/default/files/download/may_ann/onlineschedulingsystem.zip HTTP/1.1" 206 12997278 "http://www.mysite.com/php/scheduling-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0" 

顺便说一句，我testing了我的fail2ban设置使用下面的命令，它工作正常。

• 如何阻止networking爬虫下载文件
• Java 7 Server SE JRE安装
• 防止服务器无法使用WordPress的暴力攻击
• Apache 2.2不服从文件级别的cgi权限
• fastCGI PHP如何通过auditd更改文件并逃脱检测？

 fail2ban-regex /var/log/ispconfig/httpd/mysite.com/yesterday-access.log /etc/fail2ban/filter.d/apache-badbots.conf 

其实它用下面的日志检测到两个IP地址：

 5.9.23.42 - - [19/Jul/2013:07:52:26 +0800] "GET / HTTP/1.1" 200 11220 "-" "Mozilla/3.0 (compatible; Indy Library)" 124.122.67.67 - - [19/Jul/2013:02:43:30 +0800] "GET / HTTP/1.1" 200 59663 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent" 124.122.67.67 - - [19/Jul/2013:02:43:40 +0800] "-" 400 0 "-" "-" 

那么fail2ban怎么没有检测到呢？ 这真的是一个机器人吗？

我也使用cloudflare防止这种types的机器人，但我需要手动阻止IP地址或更改设置为“我受到攻击”。 这工作正常，如果我总是看我的服务器。 但是如果我离开我的电脑呢？ 那么如何防止这个爬虫呢？

请帮忙。

更新：

这是我的jail.local设置：

 [nginx-badbots] enabled = true filter = apache-badbots action = iptables-multiport[name=BadBots, port="http,https"] logpath = /var/log/ispconfig/httpd/mysite.com/access.log bantime = 86400 # 1 day maxretry = 1