shell下做字符比对输出

baolei161

请群里大神帮帮忙，能否帮我解决一下这个问题：

我要往数据库里LOAD数据，所以需要从日志文件里解析jSON格式的：
{"data":[{"position":"-1","screen":"720*1184","networkType":"WIFI","phone_number":"2222222","packageName":"aaa","channelId":"111","imei":"000","IP":"1.1.1.1","UUID":"ccc","versionCode":"20150720","mac":"00:00:00:00:00:00","agent":"infocus InFocus M310","simcardType":"ddd","linkId":"","actionDate":"2015-08-19 00:14:02","hasSDCard":"1","userId":"","name":"","sim":"33333333333","mobileOS":"4.2.2","logType":"000","resType":"-1"}]}|20150819001404|444444444
我需要的是-1，720*1184，WIFI，222222等值插入数据库数据库里对应的字段就是position，screen，networkType，phone_numbe这些，一开始我使用awk -F'"' '{for(i=6;i<=90;i=i+4) printf $i"#";printf "\n"命令将所需的值提取出来，用#号分割。但是后来发现生产的日志中有些日志缺少KEY值，就是有可能会缺少mac,sim的键值，这样用我的命令分割出来的数据中间就可能少一个，造成插入数据库的数据错位了，现在我想做的事情是，将数据库里所需要的字段与日志匹配，如果有这个字段就输出对应的值，如果没有则输出一个空值，这样保证插入的数据列数是一样的
  


请哪位大神帮帮忙给个解决办法啥的，不胜感激

MMMIX

这东西用 shell 做真不是什么好的选择.

baolei161

我当时的思路是用一个数组存所有的字段，然后用AWK分割出日志后，用数组里的值与取出的值匹配，如果相同就输出$i+2的值，如果不匹配就输出#，但是AWK命令我不太熟悉，不知道怎么把这个值和数组里面的值比较回复 2# MMMIX


   

tuyajie

awk -F'[,{[]' '{for(i=1;i<=NF;i++){split($i,a,":"); if(a[1]~"position") b[NR][1]=a[2];if(a[1]~"screen") b[NR][2]=a[2];if(a[1]~"networkType") b[NR][3]=a[2];if(a[1]~"phone_numbe") b[NR][4]=a[2]}}END{for(m=1;m<=NR;m++){for(n=1;n<=4;n++) printf b[m][n]"#";print ""}}' test.dat

baolei161

好像不行回复 4# tuyajie


   

tc1989tc

1. awk -vRS='\\[\\{|\\}\\]' -F ',' '/position/{ for (i=1;i<NF;i++)print $i;}' data.txt      
   
2. "position":"-1"
   
3. "screen":"720*1184"
   
4. "networkType":"WIFI"
   
5. "phone_number":"2222222"
   
6. "packageName":"aaa"
   
7. "channelId":"111"
   
8. "imei":"000"
   
9. "IP":"1.1.1.1"
   
10. "UUID":"ccc"
    
11. "versionCode":"20150720"
    
12. "mac":"00:00:00:00:00:00"
    
13. "agent":"infocusInFocus M310"
    
14. "simcardType":"ddd"
    
15. "linkId":""
    
16. "actionDate":"2015-08-19:00:14:02"
    
17. "hasSDCard":"1"
    
18. "userId":""
    
19. "name":""
    
20. "sim":"33333333333"
    
21. "mobileOS":"4.2.2"
    
22. "logType":"000"

复制代码

tc1989tc

1. awk -vRS='\\[\\{|\\}\\]' -F ',' '/position/{ for (i=1;i<NF;i++){ split($i,aa,":");print aa[1],aa[2] } }' data.txt
   
2. "position" "-1"
   
3. "screen" "720*1184"
   
4. "networkType" "WIFI"
   
5. "phone_number" "2222222"
   
6. "packageName" "aaa"
   
7. "channelId" "111"
   
8. "imei" "000"
   
9. "IP" "1.1.1.1"
   
10. "UUID" "ccc"
    
11. "versionCode" "20150720"
    
12. "mac" "00
    
13. "agent" "infocusInFocus M310"
    
14. "simcardType" "ddd"
    
15. "linkId" ""
    
16. "actionDate" "2015-08-19
    
17. "hasSDCard" "1"
    
18. "userId" ""
    
19. "name" ""
    
20. "sim" "33333333333"
    
21. "mobileOS" "4.2.2"
    
22. "logType" "000"

复制代码

hjfeng1988

回复 9# baolei161

1. [root@66a awk]# cat file
   
2. "a":"1","b":"2","c":"3","d":"4","e":"5"
   
3. "a":"1","b":"2","d":"4"
   
4. "a":"1","b":"2","d":"4","e":"5"
   
5. "a":"1","b":"2","c":"3","d":"4"
   
6. [root@66a awk]# awk -F ',' -vx='c":"0","' -vy=',"e":"0"' '!/"c"/{sub(/d/,x)}!/"e"/{sub(/$/,y)}{for(i=1;i<=NF;i++){split($i,a,"\"");printf a[4]}print ""}' file
   
7. 12345
   
8. 12040
   
9. 12045
   
10. 12340
    
11. [root@66a awk]#

复制代码

baolei161

貌似后面的几位大神没理解我的需求，比如 我现在数据库里有 a b c d e 共五个字段，在log中会有对应的值 a:1 b:2 c:3 d:4 e:5,但是由于客户端的LOG在记录的时候如果该字段没有获取到值的话会自动把这个KEY不记录，所以日志可能变成 a:1 b:2 d:4 这种，那么我现在的需求是我取的值必须是5个，如果发现没有则需要补一个空值，比如a:1 b:2 c:null d:4 e:null ，这样保证我loaddata进数据库的数据不会错

tuyajie

1. 
   
2. 我自己伪造的文件，第一行是没有phone_number关键字
   
3. [jay.liu@aliCloud chinaUnix]$ cat test.dat
   
4. {"data":[{"position":"-1","screen":"720*1184","networkType":"WIFI","packageName":"aaa","channelId":"111","imei":"000","IP":"1.1.1.1","UUID":"ccc","versionCode":"20150720","mac":"00:00:00:00:00:00","agent":"infocus InFocus M310","simcardType":"ddd","linkId":"","actionDate":"2015-08-19 00:14:02","hasSDCard":"1","userId":"","name":"","sim":"33333333333","mobileOS":"4.2.2","logType":"000","resType":"-1"}]}|20150819001404|444444444
   
5. {"data":[{"position":"-1","screen":"720*1184","networkType":"WIFI","phone_number":"2222222","packageName":"aaa","channelId":"111","imei":"000","IP":"1.1.1.1","UUID":"ccc","versionCode":"20150720","mac":"00:00:00:00:00:00","agent":"infocus InFocus M310","simcardType":"ddd","linkId":"","actionDate":"2015-08-19 00:14:02","hasSDCard":"1","userId":"","name":"","sim":"33333333333","mobileOS":"4.2.2","logType":"000","resType":"-1"}]}|20150819001404|444444444
   
6. {"data":[{"position":"-1","screen":"720*1184","networkType":"WIFI","phone_number":"2222222","packageName":"aaa","channelId":"111","imei":"000","IP":"1.1.1.1","UUID":"ccc","versionCode":"20150720","mac":"00:00:00:00:00:00","agent":"infocus InFocus M310","simcardType":"ddd","linkId":"","actionDate":"2015-08-19 00:14:02","hasSDCard":"1","userId":"","name":"","sim":"33333333333","mobileOS":"4.2.2","logType":"000","resType":"-1"}]}|20150819001404|444444444
   
7. 
   
8. 
   
9. 
   
10. [jay.liu@aliCloud chinaUnix]$ awk -F'[,{[]' '{for(i=1;i<=NF;i++){split($i,a,":"); if(a[1]~"position") b[NR][1]=a[2];if(a[1]~"screen") b[NR][2]=a[2];if(a[1]~"networkType") b[NR][3]=a[2];if(a[1]~"phone_numbe") b[NR][4]=a[2]}}END{for(m=1;m<=NR;m++){for(n=1;n<=4;n++) printf b[m][n]"#";print ""}}' test.dat
    
11. "-1"#"720*1184"#"WIFI"##
    
12. "-1"#"720*1184"#"WIFI"#"2222222"#
    
13. "-1"#"720*1184"#"WIFI"#"2222222"#

复制代码

回复 5# baolei161

这个是不是你要的啊？你那里抱了什么错？