用AWK统计apache响应时间最长的10条请求的URL TOP10

camwei

用AWK统计apache响应时间最长的10条请求的URL TOP10

##访问日志例子：      
1.1.1.1"[10/Dec/2011:23:30:00 +0800]"POST /t.do?requestid=apgrade HTTP/1.1"200"710001"A0001"616510220"B00001"
1.1.1.2"[10/Dec/2011:23:30:00 +0800]"GET http://*.*.*/421/10421640/l57411079840.png HTTP/1.1"200"710001"A0002"60916773"B00002"   

使用以下命令可以统计出来，但是不能去掉重复的URL，请教各位怎样能去除重复的URL？

/log01>more  access_log_001|grep  " +0800]"|awk -F\" '{print$7,$3}'|sort -nr|head -n 10
61651022 POST /t.do?requestid=apgrade HTTP/1.1
61612162 POST /t.do?requestid=apgrade HTTP/1.1
61268940 POST /t.do?requestid=apgrade HTTP/1.1
61076095 POST /t.do?requestid=apgrade HTTP/1.1
61065643 POST /t.do?requestid=apgrade HTTP/1.1
60993259 POST /t.do?requestid=apgrade HTTP/1.1
60961846 POST /t.do?requestid=apgrade HTTP/1.1
60949501 POST /t.do?requestid=apgrade HTTP/1.1
60932514 POST /t.do?requestid=apgrade HTTP/1.1
60916773 GET http://*.*.*/421/10421640/l57411079840.png HTTP/1.1

yanu

xindy@NAS ~/tmp $ cat file 61651022 POST /t.do?requestid=apgrade HTTP/1.1
61612162 POST /t.do?requestid=apgrade HTTP/1.1
61268940 POST /t.do?requestid=apgrade HTTP/1.1
61076095 POST /t.do?requestid=apgrade HTTP/1.1
61065643 POST /t.do?requestid=apgrade HTTP/1.1
60993259 POST /t.do?requestid=apgrade HTTP/1.1
60961846 POST /t.do?requestid=apgrade HTTP/1.1
60949501 POST /t.do?requestid=apgrade HTTP/1.1
60932514 POST /t.do?requestid=apgrade HTTP/1.1
60916773 GET http://*.*.*/421/10421640/l57411079840.png HTTP/1.1

xindy@NAS ~/tmp $ cat file  | sort -n -k 3,1 | awk '{a[$3]=$0}END{for(i in a){print a[i]}}'
61651022 POST /t.do?requestid=apgrade HTTP/1.1
60916773 GET http://*.*.*/421/10421640/l57411079840.png HTTP/1.1
xindy@NAS ~/tmp $


没仔细看,应该是这个意思吧?

camwei

回复 2# yanu

你这是重定向到一个文件，然后再处理，但是你这方法也不行，如下：
idt@suse10sp2:~/log> more 003.txt
60006943 POST /t.do?requestid=apgrade HTTP/1.1
60006037 POST /t.do?requestid=apgrade HTTP/1.1
34907266 GET /t.do?requestid=did=2 HTTP/1.1
33763806 GET /t.do?requestid=did=2 HTTP/1.1
31024120 GET /t.do?requestid=dontent%3ame HTTP/1.0
30586241 GET http://*.*.*m/t.do?requestid=dot&xType=nt%3e HTTP/1.1
30473203 GET http://*.*.*/t.do?requestid=dot&xType=nt:ge HTTP/1.1
30412504 GET http://*.*.*/t.do?requestid=dos&id=Cme HTTP/1.1
30397419 GET http://*.*.*/t.do?requestid=dos&id=Cre HTTP/1.1
30300715 GET http://*.*.*/t.do?requestid=dot&xType=nt1HT/1.1
30300705 GET http://*.*.*/t.do?requestid=dot&xType=nt2HT/1.1
30300615 GET http://*.*.*/t.do?requestid=dot&xType=nt3HT/1.1
30300605 GET http://*.*.*/t.do?requestid=dot&xType=nt4HT/1.1


按照你的方法，结果出来了，但顺序乱了，我想要按响应时间排序的（按第一列），而且也不能选出top10，执行结果如下：
cat 003.txt| sort -n -k 3,1 | awk '{a[$3]=$0}END{for(i in a){print a}}'
idt@suse10sp2:~/log> cat 003.txt| sort -n -k 3,1 | awk '{a[$3]=$0}END{for(i in a){print a}}'
30300615 GET http://*.*.*/t.do?requestid=dot&xType=nt3HT/1.1
34907266 GET /t.do?requestid=did=2 HTTP/1.1
30586241 GET http://*.*.*m/t.do?requestid=dot&xType=nt%3e HTTP/1.1
30300705 GET http://*.*.*/t.do?requestid=dot&xType=nt2HT/1.1
30300715 GET http://*.*.*/t.do?requestid=dot&xType=nt1HT/1.1
30473203 GET http://*.*.*/t.do?requestid=dot&xType=nt:ge HTTP/1.1
30412504 GET http://*.*.*/t.do?requestid=dos&id=Cme HTTP/1.1
60006943 POST /t.do?requestid=apgrade HTTP/1.1
31024120 GET /t.do?requestid=dontent%3ame HTTP/1.0
30397419 GET http://*.*.*/t.do?requestid=dos&id=Cre HTTP/1.1
30300605 GET http://*.*.*/t.do?requestid=dot&xType=nt4HT/1.1

而我想达到如下效果，既要选出结果，又要排好序，而且是TOP10：
60006943 POST /t.do?requestid=apgrade HTTP/1.1
34907266 GET /t.do?requestid=did=2 HTTP/1.1
31024120 GET /t.do?requestid=dontent%3ame HTTP/1.0
30586241 GET http://*.*.*m/t.do?requestid=dot&xType=nt%3e HTTP/1.1
30473203 GET http://*.*.*/t.do?requestid=dot&xType=nt:ge HTTP/1.1
30412504 GET http://*.*.*/t.do?requestid=dos&id=Cme HTTP/1.1
30397419 GET http://*.*.*/t.do?requestid=dos&id=Cre HTTP/1.1
30300715 GET http://*.*.*/t.do?requestid=dot&xType=nt1HT/1.1
30300705 GET http://*.*.*/t.do?requestid=dot&xType=nt2HT/1.1
30300615 GET http://*.*.*/t.do?requestid=dot&xType=nt3HT/1.1
   

chenyx

sort -r -k 1 003.txt  | head

camwei

回复 4# chenyx

那回到最开始的问题了，没去重复
idt@suse10sp2:~/log> sort -r -k 1 003.txt  | head -10
60006943 POST /t.do?requestid=apgrade HTTP/1.1
60006037 POST /t.do?requestid=apgrade HTTP/1.1
34907266 GET /t.do?requestid=did=2 HTTP/1.1
33763806 GET /t.do?requestid=did=2 HTTP/1.1
31024120 GET /t.do?requestid=dontent%3ame HTTP/1.0
30586241 GET http://*.*.*m/t.do?requestid=dot&xType=nt%3e HTTP/1.1
30473203 GET http://*.*.*/t.do?requestid=dot&xType=nt:ge HTTP/1.1
30412504 GET http://*.*.*/t.do?requestid=dos&id=Cme HTTP/1.1
30397419 GET http://*.*.*/t.do?requestid=dos&id=Cre HTTP/1.1
30300715 GET http://*.*.*/t.do?requestid=dot&xType=nt1HT/1.1


   

chenyx

2楼不是已经给你去重了吗?你结合2楼的改进下就行了

yanu

awk 'a[$3]{next}{a[$3]=1;print}'

去掉第三列重复的行

camwei

看来还是要用到两个重定向，分三步处理了

janlzy20

uniq 去重