每天5G的nginx日志，需要怎么分析?

yu34po

RT,要分析出每IP的访问数量和每IP的访问内容。
61.4.184.92 - - [05/Dec/2013:00:10:05 +0800] "GET /data/?areaid=329103324&type=observe&date=201312042349&appid=7fde98&key=BUyFU0GyXhzGNDIFEDMQaortggDQ= HTTP/1.1" 200 76 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1.1; MI 2S MIUI/JLB23.0)" -
61.4.184.91 - - [05/Dec/2013:00:10:05 +0800] "GET /data/?areaid=101243506&type=observe&date=201301012336&appid=7cfdf9&key=oMqeris3J3IZ3CHkbOKd06X5NYg= HTTP/1.1" 200 77 "-" "Dalvik/1.4.0 (Linux; U; Android 4.0; US900G Build/GRK39F)" -
如示例数据中
1统计出每个用户请求的数量，(用户由appid字段的值来区分)
2每个用户请求的areaid和type,
3每个IP访问的数量
每天的日志有5G或者更多，这样的数据怎么处理?
在shell版块问了，用awk.效率有点低，perl是不是会比较快一些?
awk的代码:
/appid/&&/areaid/&&/type/{
        appid=gensub(/.*appid=([^&]*).*/,"\\1",1);
        areaid=gensub(/.*areaid=([^&]*).*/,"\\1",1);
        type=gensub(/.*type=([^&]*).*/,"\\1",1);
        IP=$1;

        a[appid]++;
        if(!x[appid,areaid]++)b[appid]=b[appid]?b[appid]" "areaid:areaid;
        if(!y[appid,type]++)c[appid]=c[appid]?c[appid]" "type:type;
        d[IP]++;
}

END{
        for(i in a)printf "appid:\t%s\ntimes:\t%d\nareaid:\t%s\ntype:\t%s\n\n",i,a[i],b[i],c[i];
        for(i in d)print i,d[i];
}

yu34po

回复 36# rubyish

太牛了。。

rubyish

biru:

1. #!/usr/bin/perl
   
2. 
   
3. my $log = 'mytestlog';
   
4. open my $f, $log or die "$log: $!";
   
5. my ( %ip, %id, %url );
   
6. 
   
7. while (<$f>) {
   
8.     my $url = ( split /"/, $_, 3 )[1];
   
9.     $url = substr $url, 4;
   
10.     next unless $url;
    
11.     my ( $ip, $appid ) = /^(\S+).*?appid=(\w+)/;
    
12.     next unless $appid;
    
13.     $url{$url}{times}++;
    
14.     $ip{$ip}{times}++;
    
15.     $id{$appid}{times}++;
    
16.     $ip{$ip}{url}{$url}       = 1;
    
17.     $ip{$ip}{appid}{$appid}   = 1;
    
18.     $url{$url}{ip}{$ip}       = 1;
    
19.     $url{$url}{appid}{$appid} = 1;
    
20.     $id{$appid}{url}{$url}    = 1;
    
21.     $id{$appid}{ip}{$ip}      = 1;
    
22. }
    
23. 
    
24. while ( my ( $k, $v ) = each %id ) {
    
25.     next if length $k != 6;
    
26.     print "appid:\t", $k, $/;
    
27.     print "times:\t", $v->{times}, $/;
    
28.     print "url:\t", join( ', ', keys %{ $v->{url} } ), $/;
    
29.     print "ip:\t", join( ', ', keys %{ $v->{ip} } ), $/, $/;
    
30. }
    
31. 
    
32. print $/;
    
33. 
    
34. while ( my ( $k, $v ) = each %ip ) {
    
35.     next if $v->{times} < 1000;
    
36.     print "ip:\t$k$/";
    
37.     print "times:\t$v->{times}$/";
    
38.     print "url:\t", join( ', ', keys %{ $v->{url} } ), $/;
    
39.     print "appid:\t", join( ', ', keys %{ $v->{appid} } ), $/, $/;
    
40. }
    
41. 
    
42. print $/;
    
43. 
    
44. while ( my ( $k, $v ) = each %url ) {
    
45.     next if $v->{times} < 1000;
    
46.     print "url:\t$k$/";
    
47.     print "times:\t$v->{times}$/";
    
48.     print "ip:\t", join( ', ', keys %{ $v->{ip} } ), $/;
    
49.     print "appid:\t", join( ', ', keys %{ $v->{appid} } ), $/, $/;
    
50. }
    

复制代码

yu34po

1111111111111111111111

yu34po

回复 31# rubyish


    大神快来啊

yu34po

回复 31# rubyish


    一条ip可能对应多条appid和多条ref,这还真难弄了。

yu34po

回复 31# rubyish

61.4.184.92 - - [05/Dec/2013:00:10:05 +0800] "GET /data/?areaid=101091101&type=observe&date=201312042349&appid=74uq29&key=BfefdfdhzGNDNVpMQaortggDQ= HTTP/1.1" 200 76 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1.1; MI 2S MIUI/JLB23.0)" -
61.4.184.90 - - [05/Dec/2013:00:10:06 +0800] "GET /data/?areaid=101210101&type=forecast&date=201312042349&appid=f61f32&key=Cums98rfefesdmJA85YDaOO4DEG74%3D HTTP/1.1" 200 1167 "-" "SAMSUNG-Android"
我去，这下完了，说要统计以appid为key对应的ip,url和请求的次数，还要以ip为key，对应的appid，url和请求的次数，再要以url为key,对应的appid，ip和请求的次数，
我去……。adkh
appid:    d30b9f//以appid为key
times:    30
url:        "/data/?areaid=101091101&type=observe&date=201312042349&appid=74uq29&key=BfefdfdhzGNDNVpMQaortggDQ= HTTP/1.1","..."
ip:         "10.10.10.10","11.11.11.11","..."

ip:         "1.1.1.1"//以ip为key,只统计超过1000条的数据
times:    300
url:        "/data/?areaid=101091101&type=observe&date=201312042349&appid=74uq29&key=BfefdfdhzGNDNVpMQaortggDQ= HTTP/1.1","..."
appid:    "d30b9f"

url:        "/data/?areaid=101091101&type=observe&date=201312042349&appid=74uq29&key=BfefdfdhzGNDNVpMQaortggDQ= HTTP/1.1"//以url为  
             key，只统计超过1000条的数据
times:    40
ip:         "10.10.10.11","..."
appid:    "d30b9f","..."

rubyish

现在主要的问题是url是某段?~

61.4.184.92 - - [05/Dec/2013:00:13:54 +0800] "GET /data/?areaid=101211001&type=observe&date=201312042352&appid=f63d32&key=hktQjJJME8FVe1RKC6cg2vIfh3M%3D HTTP/1.1" 200 297 "-" "SAMSUNG-Android" 10.239.163.3
61.4.184.90 - - [05/Dec/2013:00:13:54 +0800] "GET /data/?areaid=101080801&type=observe&date=201312042352&appid=7c1429&key=cLX3kjDH4xin7TeXbA1jTfbKHnw= HTTP/1.1" 200 77 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1.2; Coolpad 7268 Build/JZO54K)" -

yu34po

回复 29# pitonas


擦，url就是url,这个需求我还纳闷呢，是要把每个url都给统计出来
   

pitonas

大牛，where is 这个url的访问的url次数？{:2_172:}