Chinaunix
标题:
shell处理json文件
[打印本页]
作者:
guobaofu
时间:
2019-08-19 22:58
标题:
shell处理json文件
cu的朋友们请帮忙,
json文件我是用curl每分钟在我们spring boot 微服务上去取。
需要执行效率高些,因为打算往日志服务器上传(logstash->es)。
json文件是嵌套了多个数组,数组数量不是固定的,是动态变化的。以下面这个json文件为例子包含了四个数组 “
msgprocessors”:[],"
endpoints":[],"
services":[],"
apis":[].
我的要求是如何将这个json文件里每个数组的内容取出来,放在json的{}里,每一个数组占一行,有多少数组就有多少行。
json 源文件
{
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
}
],
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
}
],
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
}
],
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
}
]
}
复制代码
需要输出样例文件如下(json数组数量不固定的,当前例子就是四个数组):
第一行:
{
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
}
]
}
第二行:
{
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
}
]
}
第三行:
{
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
}
]
}
第四行:
{
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
}
]
}
复制代码
作者:
Shell_HAT
时间:
2019-08-20 08:26
试过jq命令吗?
作者:
cfwyy
时间:
2019-08-20 15:10
本帖最后由 cfwyy 于 2019-08-21 09:01 编辑
感觉正规路子应该用jq,
不过jq太强了,现学还没怎么会用
先来个野路子 sed
$ sed -E 's/(\]),/\1\n\}\n\{/' json.txt
复制代码
用jq的话,jq本身的循环遍历没怎么搞懂,不知道能不能实现。
先用shell的循环来调jq,就是效率不高。
#!/bin/bash
jsonFile="json.txt"
for i in `jq 'keys[]' $jsonFile` #先取得属性名字
do
k=`echo $i|tr -d '"'` #去掉引号,否则会带引号传给jq
jq --arg t $k '{($t):.[$t]}' $jsonFile # 按每个属性名:对应属性值 单独构造成json对象。
done
复制代码
作者:
guobaofu
时间:
2019-08-21 15:14
本帖最后由 guobaofu 于 2019-08-21 15:20 编辑
回复
3#
cfwyy
hi,我测试过你的sed命令和脚本,对于我上面的源文件输出可以,但是我的json文件内容比那个多,我测试输出有问题。
json.txt
{
"Components": {
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
},
{
"processor": "ch-retry-outbound-sampling",
"active": true
},
{
"processor": "pt-retry-outbound-sampling",
"active": true
},
{
"processor": "be-biometrics-inbox-processor",
"active": true
},
{
"processor": "pt-biometrics-inbox-processor",
"active": true
},
{
"processor": "ch-biometrics-inbox-processor",
"active": true
},
{
"processor": "uk-biometrics-inbox-processor",
"active": true
},
{
"processor": "eshop-retryAppointment-fail-processor",
"active": true
},
{
"processor": "be-retry-outbound-sampling",
"active": true
},
{
"processor": "uk-retry-outbound-sampling",
"active": true
},
{
"processor": "eshop-appointment-processor",
"active": true
},
{
"processor": "uk-priority-retry-outbound-sampling",
"active": true
}
],
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
},
{
"endpoint": "be-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "ch-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "drools_antifraud",
"active": true
},
{
"endpoint": "drools_appointment_lateness",
"active": true
},
{
"endpoint": "drools_bioqueue",
"active": true
},
{
"endpoint": "drools_dynamic_endpoint",
"active": true
},
{
"endpoint": "endpoint-admin-serviceEP",
"active": true
},
{
"endpoint": "eshop-out-appointment-endpoint",
"active": true
},
{
"endpoint": "eshop-out-order-endpoint",
"active": true
},
{
"endpoint": "eshop_token_endpoint",
"active": true
},
{
"endpoint": "lb-be-encrypt-endpoint",
"active": true
},
{
"endpoint": "msgprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "pt-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "queue-admin-serviceEP",
"active": true
},
{
"endpoint": "servicegroup-admin-serviceEP",
"active": true
},
{
"endpoint": "tlsconnect_appointment_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_order_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_tma",
"active": true
},
{
"endpoint": "uk-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "uk-hone1adapter-outbound-endpoint",
"active": true
}
],
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
},
{
"service": "uk-filedecrypt-service",
"active": true
},
{
"service": "uk-hone1adapter-outbound-service",
"active": true
},
{
"service": "Version",
"active": true
}
],
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
},
{
"api": "order",
"active": true
},
{
"api": "be_token",
"active": true
},
{
"api": "GenericRulesEngineAPI",
"active": true
},
{
"api": "tma",
"active": true
},
{
"api": "sre",
"active": true
},
{
"api": "esbHealthCheckAPI",
"active": true
}
]
}
}
复制代码
cat json.txt|sed -E 's/(\]),/\1\n\}\n\{/'
输出结果:
{
"Components": {
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
},
{
"processor": "ch-retry-outbound-sampling",
"active": true
},
{
"processor": "pt-retry-outbound-sampling",
"active": true
},
{
"processor": "be-biometrics-inbox-processor",
"active": true
},
{
"processor": "pt-biometrics-inbox-processor",
"active": true
},
{
"processor": "ch-biometrics-inbox-processor",
"active": true
},
{
"processor": "uk-biometrics-inbox-processor",
"active": true
},
{
"processor": "eshop-retryAppointment-fail-processor",
"active": true
},
{
"processor": "be-retry-outbound-sampling",
"active": true
},
{
"processor": "uk-retry-outbound-sampling",
"active": true
},
{
"processor": "eshop-appointment-processor",
"active": true
},
{
"processor": "uk-priority-retry-outbound-sampling",
"active": true
}
]
}
{
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
},
{
"endpoint": "be-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "ch-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "drools_antifraud",
"active": true
},
{
"endpoint": "drools_appointment_lateness",
"active": true
},
{
"endpoint": "drools_bioqueue",
"active": true
},
{
"endpoint": "drools_dynamic_endpoint",
"active": true
},
{
"endpoint": "endpoint-admin-serviceEP",
"active": true
},
{
"endpoint": "eshop-out-appointment-endpoint",
"active": true
},
{
"endpoint": "eshop-out-order-endpoint",
"active": true
},
{
"endpoint": "eshop_token_endpoint",
"active": true
},
{
"endpoint": "lb-be-encrypt-endpoint",
"active": true
},
{
"endpoint": "msgprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "pt-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "queue-admin-serviceEP",
"active": true
},
{
"endpoint": "servicegroup-admin-serviceEP",
"active": true
},
{
"endpoint": "tlsconnect_appointment_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_order_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_tma",
"active": true
},
{
"endpoint": "uk-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "uk-hone1adapter-outbound-endpoint",
"active": true
}
]
}
{
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
},
{
"service": "uk-filedecrypt-service",
"active": true
},
{
"service": "uk-hone1adapter-outbound-service",
"active": true
},
{
"service": "Version",
"active": true
}
]
}
{
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
},
{
"api": "order",
"active": true
},
{
"api": "be_token",
"active": true
},
{
"api": "GenericRulesEngineAPI",
"active": true
},
{
"api": "tma",
"active": true
},
{
"api": "sre",
"active": true
},
{
"api": "esbHealthCheckAPI",
"active": true
}
]
}
}
复制代码
jq输出有问题啊
cat json-out.txt |jq -c
parse error: Expected separator between values at line 58, column 1
希望输出结果是这样子:
{
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
},
{
"processor": "ch-retry-outbound-sampling",
"active": true
},
{
"processor": "pt-retry-outbound-sampling",
"active": true
},
{
"processor": "be-biometrics-inbox-processor",
"active": true
},
{
"processor": "pt-biometrics-inbox-processor",
"active": true
},
{
"processor": "ch-biometrics-inbox-processor",
"active": true
},
{
"processor": "uk-biometrics-inbox-processor",
"active": true
},
{
"processor": "eshop-retryAppointment-fail-processor",
"active": true
},
{
"processor": "be-retry-outbound-sampling",
"active": true
},
{
"processor": "uk-retry-outbound-sampling",
"active": true
},
{
"processor": "eshop-appointment-processor",
"active": true
},
{
"processor": "uk-priority-retry-outbound-sampling",
"active": true
}
]
}
{
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
},
{
"endpoint": "be-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "ch-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "drools_antifraud",
"active": true
},
{
"endpoint": "drools_appointment_lateness",
"active": true
},
{
"endpoint": "drools_bioqueue",
"active": true
},
{
"endpoint": "drools_dynamic_endpoint",
"active": true
},
{
"endpoint": "endpoint-admin-serviceEP",
"active": true
},
{
"endpoint": "eshop-out-appointment-endpoint",
"active": true
},
{
"endpoint": "eshop-out-order-endpoint",
"active": true
},
{
"endpoint": "eshop_token_endpoint",
"active": true
},
{
"endpoint": "lb-be-encrypt-endpoint",
"active": true
},
{
"endpoint": "msgprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "pt-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "queue-admin-serviceEP",
"active": true
},
{
"endpoint": "servicegroup-admin-serviceEP",
"active": true
},
{
"endpoint": "tlsconnect_appointment_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_order_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_tma",
"active": true
},
{
"endpoint": "uk-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "uk-hone1adapter-outbound-endpoint",
"active": true
}
]
}
{
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
},
{
"service": "uk-filedecrypt-service",
"active": true
},
{
"service": "uk-hone1adapter-outbound-service",
"active": true
},
{
"service": "Version",
"active": true
}
]
}
{
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
},
{
"api": "order",
"active": true
},
{
"api": "be_token",
"active": true
},
{
"api": "GenericRulesEngineAPI",
"active": true
},
{
"api": "tma",
"active": true
},
{
"api": "sre",
"active": true
},
{
"api": "esbHealthCheckAPI",
"active": true
}
]
}
复制代码
cat json-out.txt |jq -c
{"msgprocessors":[{"processor":"eshop-retryOrder-fail-processor","active":true},{"processor":"eshop-order-processor","active":true},{"processor":"ch-retry-outbound-sampling","active":true},{"processor":"pt-retry-outbound-sampling","active":true},{"processor":"be-biometrics-inbox-processor","active":true},{"processor":"pt-biometrics-inbox-processor","active":true},{"processor":"ch-biometrics-inbox-processor","active":true},{"processor":"uk-biometrics-inbox-processor","active":true},{"processor":"eshop-retryAppointment-fail-processor","active":true},{"processor":"be-retry-outbound-sampling","active":true},{"processor":"uk-retry-outbound-sampling","active":true},{"processor":"eshop-appointment-processor","active":true},{"processor":"uk-priority-retry-outbound-sampling","active":true}]}
{"endpoints":[{"endpoint":"activeprocessor-admin-serviceEP","active":true},{"endpoint":"api-admin-serviceEP","active":true},{"endpoint":"be-filedecrypt-service-endpoint","active":true},{"endpoint":"ch-filedecrypt-service-endpoint","active":true},{"endpoint":"drools_antifraud","active":true},{"endpoint":"drools_appointment_lateness","active":true},{"endpoint":"drools_bioqueue","active":true},{"endpoint":"drools_dynamic_endpoint","active":true},{"endpoint":"endpoint-admin-serviceEP","active":true},{"endpoint":"eshop-out-appointment-endpoint","active":true},{"endpoint":"eshop-out-order-endpoint","active":true},{"endpoint":"eshop_token_endpoint","active":true},{"endpoint":"lb-be-encrypt-endpoint","active":true},{"endpoint":"msgprocessor-admin-serviceEP","active":true},{"endpoint":"pt-filedecrypt-service-endpoint","active":true},{"endpoint":"queue-admin-serviceEP","active":true},{"endpoint":"servicegroup-admin-serviceEP","active":true},{"endpoint":"tlsconnect_appointment_endpoint","active":true},{"endpoint":"tlsconnect_order_endpoint","active":true},{"endpoint":"tlsconnect_tma","active":true},{"endpoint":"uk-filedecrypt-service-endpoint","active":true},{"endpoint":"uk-hone1adapter-outbound-endpoint","active":true}]}
{"services":[{"service":"pt-filedecrypt-service","active":true},{"service":"uk-file-processor-service","active":true},{"service":"uk-filedecrypt-service","active":true},{"service":"uk-hone1adapter-outbound-service","active":true},{"service":"Version","active":true}]}
{"apis":[{"api":"appointmentLateness","active":true},{"api":"bioqueue","active":true},{"api":"pt-emetrics_TLS2PortugalAPI","active":true},{"api":"order","active":true},{"api":"be_token","active":true},{"api":"GenericRulesEngineAPI","active":true},{"api":"tma","active":true},{"api":"sre","active":true},{"api":"esbHealthCheckAPI","active":true}]}
复制代码
作者:
cfwyy
时间:
2019-08-21 16:14
回复
4#
guobaofu
你现在的例子 和你一开始给出的例子 不是内容多和少的问题,是结构不一样,当然直接用肯定有问题了。
我sed的想法就是刚好按你一开始的结构,简单的做两个括号的替换就可以了。
可以联合jq稍改一下。
jq '.[]' 2.txt |sed -E 's/(\]),/\1\n\}\n\{/'
复制代码
脚本的方法根据新的json结构也可以改一下:
#!/bin/bash
jsonFile="json.txt"
for i in `jq '.[]|keys[]' $jsonFile`
do
k=`echo $i|tr -d '"'`
jq --arg t $k '{($t):.[][$t]}' $jsonFile
done
复制代码
作者:
本友会机友会摄友会
时间:
2019-08-27 14:31
提示:
作者被禁止或删除 内容自动屏蔽
作者:
guobaofu
时间:
2019-08-29 13:32
回复
6#
本友会机友会摄友会
多谢兄台相助,好用了。
作者:
guobaofu
时间:
2019-08-29 13:33
回复
5#
cfwyy
仁兄,你的我已经测试了,妥妥的好用。
欢迎光临 Chinaunix (http://bbs.chinaunix.net/)
Powered by Discuz! X3.2