免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
查看: 9537 | 回复: 8

[文本处理] 请用一行过滤出文件内容,然后输出json [复制链接]

论坛徽章:
8
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015小元宵徽章
日期:2015-03-06 15:58:18每日论坛发贴之星
日期:2015-06-08 22:20:00每日论坛发贴之星
日期:2015-06-08 22:20:00操作系统版块每日发帖之星
日期:2015-06-14 22:20:00数据库技术版块每日发帖之星
日期:2015-11-09 06:20:00数据库技术版块每日发帖之星
日期:2016-02-22 06:20:0015-16赛季CBA联赛之上海
日期:2017-01-01 23:58:53
发表于 2020-07-28 15:05 |显示全部楼层
想过滤日志文件,日志文件是nginx的json格式,大家可以用jq命令使用。现在我想用一行,来过滤出数据得到结果,然后让zabbix去自动发现。
原始日志如下:


  1. {  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.265", "upstream_response_time": "-", "body_bytes_sent":"355289", "request": "GET /js/chunk-6b0d4374.62b34cbc.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-6b0d4374.62b34cbc.js" }
  2. {  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"1371", "request": "GET /js/chunk-dc4422f4.344bfc79.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-dc4422f4.344bfc79.js" }
  3. {  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.330", "upstream_response_time": "-", "body_bytes_sent":"380267", "request": "GET /img/login-bg.846ea57d.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/login-bg.846ea57d.png" }
  4. {  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"19041", "request": "GET /img/no-data.72bde197.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/no-data.72bde197.png" }
  5. {  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"12059", "request": "GET /img/QRCode-icon.c1207a95.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/QRCode-icon.c1207a95.png" }
复制代码
想要的结果是输出如下的内容:
  1. {"num":"5","ip":"192.168.3.6"}
复制代码
输出的结果是一段json,我们通过jq也可以看出是json
  1. $ echo '{"num":"5","ip":"192.168.3.6"}' |jq .
  2. {
  3.   "num": "5",
  4.   "ip": "192.168.3.6"
  5. }
复制代码



论坛徽章:
0
发表于 2020-08-09 13:01 |显示全部楼层
[code]# cat /tmp/1
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.265", "upstream_response_time": "-", "body_bytes_sent":"355289", "request": "GET /js/chunk-6b0d4374.62b34cbc.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-6b0d4374.62b34cbc.js" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"1371", "request": "GET /js/chunk-dc4422f4.344bfc79.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-dc4422f4.344bfc79.js" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.330", "upstream_response_time": "-", "body_bytes_sent":"380267", "request": "GET /img/login-bg.846ea57d.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/login-bg.846ea57d.png" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"19041", "request": "GET /img/no-data.72bde197.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/no-data.72bde197.png" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"12059", "request": "GET /img/QRCode-icon.c1207a95.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/QRCode-icon.c1207a95.png" }

# awk '{ip = gensub(/\"(.+)\",/, "\\1", "g", $6); count[ip]++} END{for (i in count) {printf "{\"num\":\"%d\", \"ip\":\"%s\"}\n", count[ip],ip}}' /tmp/1
{"num":"5", "ip":"117.136.39.219"}
[/code]

论坛徽章:
0
发表于 2020-08-09 13:02 |显示全部楼层

[root@mybox mail]# cat /tmp/1
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.265", "upstream_response_time": "-", "body_bytes_sent":"355289", "request": "GET /js/chunk-6b0d4374.62b34cbc.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-6b0d4374.62b34cbc.js" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"1371", "request": "GET /js/chunk-dc4422f4.344bfc79.js HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/js/chunk-dc4422f4.344bfc79.js" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.330", "upstream_response_time": "-", "body_bytes_sent":"380267", "request": "GET /img/login-bg.846ea57d.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/login?sourceType=hlldriver&revision=5947&client_type=android_app&token=7858713c03724b89b43d2b93d017f367&model=CDY-AN00&brand=HUAWEI", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/login-bg.846ea57d.png" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"19041", "request": "GET /img/no-data.72bde197.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/no-data.72bde197.png" }
{  "timestamp": "28/Jul/2020:14:58:55 +0800", "remote_addr": "117.136.39.219", "status": "200", "request_time": "0.000", "upstream_response_time": "-", "body_bytes_sent":"12059", "request": "GET /img/QRCode-icon.c1207a95.png HTTP/1.1", "http_referrer": "https://charge.huolala.cn/", "upstream_addr": "-", "http_x_real_ip": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Linux; Android 10; CDY-AN00 Build/HUAWEICDY-AN00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/77.0.3865.120 MQQBrowser/6.2 TBS/045230 Mobile Safari/537.36/huolala(5947,android,SZ-S-A-Y-huawei,5.9.47,CDY-AN00,af5aea1a816c7d278e36de1afb752c99,)","request_filename": "/home/data/webroot/charge.huolala.cn/dist/img/QRCode-icon.c1207a95.png" }
[root@mybox mail]#
[root@mybox mail]#
[root@mybox mail]# awk '{ip = gensub(/\"(.+)\",/, "\\1", "g", $6); count[ip]++} END{for (i in count) {printf "{\"num\":\"%d\", \"ip\":\"%s\"}\n", count[ip],ip}}' /tmp/1
{"num":"5", "ip":"117.136.39.219"}

论坛徽章:
0
发表于 2020-08-11 16:06 |显示全部楼层
提示: 作者被禁止或删除 内容自动屏蔽

论坛徽章:
1
19周年集字徽章-年
日期:2020-10-29 09:39:21
发表于 2020-08-19 09:32 |显示全部楼层
本帖最后由 legs 于 2020-08-20 08:20 编辑
  1. perl -nE'/addr": "([^"]+)/&&$h{$1}++}{say"{\"num\":\"$h{$_}\",\"ip\":\"$_\"}"for keys%h'
复制代码

论坛徽章:
1
19周年集字徽章-年
日期:2020-10-29 09:39:21
发表于 2020-08-19 09:33 |显示全部楼层
  1. perl -nE'/addr": "([^"]+)/&&$h{$1}++}{say"{\"num\":\"$h{$_}\",\"ip\":\"$_\"}"for keys%h'
复制代码

论坛徽章:
1
19周年集字徽章-年
日期:2020-10-29 09:39:21
发表于 2020-08-19 09:33 |显示全部楼层
perl -nE'/addr": "([^"]+)/&&$h{$1}++}{say"{\"num\":\"$h{$_}\",\"ip\":\"$_\"}"for keys%h'

论坛徽章:
2
15-16赛季CBA联赛之四川
日期:2020-08-18 20:19:4815-16赛季CBA联赛之青岛
日期:2022-01-19 11:51:49
发表于 2020-08-19 15:14 |显示全部楼层
legs 发表于 2020-08-19 09:33
perl -nE'/addr": "([^"]+)/&&$h{$1}++}{say"{\"num\":\"$h{$_}\",\"ip\":\"$_\"}"for keys%h'

大佬,请过来帮我看看

论坛徽章:
59
2015七夕节徽章
日期:2015-08-24 11:17:25ChinaUnix专家徽章
日期:2015-07-20 09:19:30每周论坛发贴之星
日期:2015-07-20 09:19:42ChinaUnix元老
日期:2015-07-20 11:04:38荣誉版主
日期:2015-07-20 11:05:19巳蛇
日期:2015-07-20 11:05:26CU十二周年纪念徽章
日期:2015-07-20 11:05:27IT运维版块每日发帖之星
日期:2015-07-20 11:05:34操作系统版块每日发帖之星
日期:2015-07-20 11:05:36程序设计版块每日发帖之星
日期:2015-07-20 11:05:40数据库技术版块每日发帖之星
日期:2015-07-20 11:05:432015年辞旧岁徽章
日期:2015-07-20 11:05:44
发表于 2020-10-28 12:38 |显示全部楼层
脚本唯一不好的地方是太乱套了。看不明白逻辑。
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP