awk 处理

一棵菠菜

这里有50多W个ip，如果前3位是相同的就取第1个。

1. 14.221.246.118
   
2. 14.222.67.130
   
3. 27.18.178.159
   
4. 58.16.249.53
   
5. 58.22.113.5
   
6. 58.22.113.28
   
7. 58.22.116.143
   
8. 58.22.116.145
   
9. 61.158.153.216
   
10. 112.4.2.17
    
11. 112.4.2.18
    
12. 112.4.2.48
    
13. 112.4.2.49
    
14. 112.4.2.50
    
15. 112.4.2.51
    
16. 112.96.30.233
    
17. 112.96.66.5
    
18. 112.96.66.18
    
19. 112.96.128.106
    
20. 112.96.130.12
    
21. 112.96.254.55
    
22. 112.96.255.35
    
23. 112.97.30.2

复制代码

处理后

1. 14.221.246.118
   
2. 14.222.67.130
   
3. 27.18.178.159
   
4. 58.16.249.53
   
5. 58.22.113.5
   
6. 58.22.116.143
   
7. 61.158.153.216
   
8. 112.4.2.17
   
9. 112.96.30.233
   
10. 112.96.66.5
    
11. 112.96.128.106
    
12. 112.96.130.12
    
13. 112.96.254.55
    
14. 112.96.255.35
    
15. 112.97.30.2

复制代码

该如何操作呢

Shell_HAT

1. awk -F. '{if(a[$1.$2.$3]==0)a[$1.$2.$3]=$0}END{for(i in a)print a[i]}' urfile

复制代码

Shell_HAT

1. awk -F. '{if(a[$1.$2.$3]==0)a[$1.$2.$3]=$0}END{for(i in a)print a[i]}' urfile

复制代码

一棵菠菜

谢谢，就是长了点。。。

jiejie455

awk -vFS="." '{line=$1"."$2"."$3;if(a[line]==""){a[line]=$0}}END{for( i in a){ print a[i]}}' data | sort -t . -k1,1n -k2,2n -k3,3n

ly5066113

回复 4# 一棵菠菜


awk -F. '!a[$1.$2.$3]++' urfile

cjaizss

awk 'BEGIN{FS="\."}NF==4{if(a[$1,$2,$3]=="")a[$1,$2,$3]=$0}END{for(i in a)print a[i]}'

Shell_HAT

回复 4# 一棵菠菜


几厘米的算短？
希望你能懂6楼

一棵菠菜

6楼的代码很经典。学习了。谢谢大家的帮助！！！

BangBull

1. 
   
2. #!/usr/bin/python
   
3. #coding:utf-8
   
4. ipEndNumList = []
   
5. ipDict = {}
   
6. dataFile = 'ipdata.txt'
   
7. 
   
8. def ipData(dataFile):
   
9.     f = open(dataFile)
   
10.     for line in f:
    
11.         data = line.strip().rsplit('.',1)
    
12.         yield data
    
13.                
    
14. ipData = ipData(dataFile)
    
15. 
    
16. for i in ipData:
    
17.     ipSection = i[0]
    
18.     if ipSection in ipDict:
    
19.         ipDict[ipSection].append(i)
    
20.     else:
    
21.         ipDict[ipSection] = [i]
    
22. 
    
23. for keys in ipDict:
    
24.     for ipList in ipDict[keys]:
    
25.         ipEndNumList.append(int(ipList[1]))
    
26.     minNum = min(ipEndNumList)
    
27.     print keys + '.' + str(minNum)
    
28.     minNum = 0
    
29.     ipEndNumList = []
    

复制代码