- 论坛徽章:
- 0
|
本帖最后由 avyou 于 2014-04-10 17:01 编辑
nginx 服务器突然不断出现大量的 bingBot 和 Googlebot 日志。是很多不断出现哦,带宽严重上升,不是一般的蜘蛛普通抓取。
日志内容如下,其中 11.11.11.11 为我服务器的外网IP,- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 502 166 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 502 166 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 502 166 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 502 166 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -
- 11.11.11.11 - - [10/Apr/2014:16:43:35 +0800] "GET /ting/ycdxw/180.html HTTP/1.0" 504 176 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
复制代码 我根本找不到外连的IP,看到的只是自己的外网IP,如:- # netstat -ntu | awk '{print $5}' | egrep -o "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | sort | uniq -c | sort -nr |more
- 17175 127.0.0.1
- 8020 11.11.11.11
- 186 192.168.14.2
- .....
复制代码 我在nginx.conf 设置了:- if ($http_user_agent ~* "http://www.bing.com/bingbot.htm"){return 403;}
- if ($http_user_agent ~* "http://www.google.com/bot.html"){return 403;}
复制代码 访问日志才停止。
$http_user_agent 有可能是伪造的,因为我们网站需要搜索引擎收录,又不能永远过滤它,不知道如何办,各位有没有出现这种情况,是不是真的被攻击了??求助啊,谢谢各位了。 |
|