nginx_access日志处理

jgrlj322

如题,求一个shell.日志格式如下
100.116.128.29 - - [31/Aug/2017:03:32:57 +0800] "GET /ccc/index/js/main/index-29560643.js HTTP/1.0" 200 15864 "https://www.test.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" "42.156.136.10" www.test.com - - - 0.000

100.116.128.83 - - [31/Aug/2017:08:15:43 +0800] "OST /login/ajax HTTP/1.0" 200 114 "https://www.test.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 LBBROWSER" "113.235.12.33" www.test.com 127.0.0.1:8000 200 0.024 0.024


要求
输出第一个列:100.116.128.83
输出第二列:31/Aug/2017:08:15:43 +0800
输出第三列请求类型OST/GET
输出第四列状态码:200
输出第五列网站地址:https://www.test.com/
输出第七列浏览器类型:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
输出第八列访问实际IP:113.235.12.33
并
统计实际IP访问次数.
求大神给写一个最好能像表格一样显示出来的.
cat access.log-20170901 | awk '{print $1,substr($4,2,20),substr($6,2,3),$9,$11,$12,$21,$22,$23,$24}'  |sort |uniq -c |sort -k1 -nr
没有统

jgrlj322

自己写出来了.只是还需要细化很多东西
^(?<remote_addr>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(?<time_local>.*)\] "(?<request>[^"]*)" (?<status>\d+) (?<body_bytes_sent>\d+) "(?<http_referer>[^"]*)" "(?<http_user_agent>[^"]*)" "(?<http_x_forwarded_for>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" (?<http_host>(\w+\.){2}\w+) (?<upstream_addr>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5})(?<upstream_status>\d{1,3}) (?<upstream_response_time>([0-9\" ".]*))

wh7211

回复 1# jgrlj322

1. awk '{y="^([^ ]+)[^[]+\\[([^]]+)[^\"]+\"([^ ]+)[^\"]+[^0-9]+([^ ]+)[^\"]+\"([^\"]+)\"[^\"]+\"([^\"]+)\"[^\"]+\"([^\"]+)"};match($0,y,t){print t[1],t[2],t[3],t[4],t[5],t[6],t[7];a[t[7]]++}END{for(i in a){print i,a[i]}}' access.log-20170901

复制代码

jason680

回复 1# jgrlj322

How about this way...

$ awk -f get_log.awk a.log
100.116.128.29    31/Aug/2017:03:32:57 +0800    GET    200    https://www.test.com/    Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36    42.156.136.10
                        
100.116.128.83    31/Aug/2017:08:15:43 +0800    POST    200    https://www.test.com/    Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 LBBROWSER    113.235.12.33


$ cat get_log.awk
function qq( c,n){
  c = 0;
  for (n = 1; n <= NF; n++) {
    if($n~/^"/ && $n~/"$/){
      #gsub(/"/,"")  # for removed the "
      $(++c) = $n
      continue
    }
    if(q == 1){
      if($n~/"$/){
      #if(sub(/"$/,"",$n)){   # for removed the "
        q = 0
      }
      $c = $c FS $n
      continue
    }
    if($n~/^"/){
    #if(sub(/^"/,"",$n)){   # for removed the "
      q = 1
      $(++c) = $n
      continue
    }
    $(++c) = $n
  }
  NF = c
}
{
  #print "NF="NF,"before modified"
  m=$6;
  sub("\"","",m);
  qq();
  #print "NF="NF,"after modified"
  for (n = 1; n <= NF; n++) {
    gsub("\"","",$n);
    gsub("^\\[|]$","",$n);
    #printf("$%d = <%s>\n", n, $n)
  }
  ip=$1;
  time=$4" "$5
  method = m;
  status=$7
  www=$9
  broswer=$10
  real_ip=$11
  OFS="\t";
  print ip,time,method,status,www,broswer,real_ip
}

jgrlj322

非常感谢两位的大神的回复,谢谢
当初写的时候是想通过awk等工具来实现.后来用ELK发现画图狠屌.所以我写的是grok正则规则.