截取网址

方兆国儿

方兆国儿

1. 
   

复制代码

方兆国儿

请删掉多余的帖子

方兆国儿

[ 本帖最后由 方兆国儿 于 2016-12-06 12:37 编辑 ]\n\n请删除多余的帖子

方兆国儿

换浏览器还是这样，提交代码就报错！

方兆国儿

1. 请删除多余帖子

复制代码

方兆国儿

回复 2# 王楠w_n

请帮忙处理一下，谢谢。

方兆国儿

1. zip -r cu.zip cu
   
2. adding: cu/ (stored 0%)
   
3. adding: cu/cu.sh (deflated 57%)
   
4. adding: cu/cu.txt (deflated 79%)

复制代码

cu.zip (1.99 KB, 下载次数: 19)

2016-12-06 12:36 上传

点击文件名下载附件

  为方便大家，相关内容已压入附件，多谢。
大家都强调我的数据类型不完整，在此重新统计，以下文为例：

1. cat cu.txt
   
2. Dec 2 10:12:19 ooxx-term[2060]: ooxx_pool/s1 200 {Mac+OS+X/10.12.1 (16B2555) CalendarAgent/384} "CONNECT p11-caldav.icloud.com:443 HTTP/1.1"
   
3. Dec 2 10:12:21 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://www.ipip.net/ HTTP/1.1"
   
4. Dec 2 10:12:23 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT wx.qq.com:443 HTTP/1.1"
   
5. Dec 2 10:12:25 ooxx-term[2060]: ooxx_pool/s1 200 {trustd (unknown version) CFNetwork/807.1.3 Darwin/16.1.0 (x86_64)} "GET http://gn.symcd.com/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFLGLCwGXUwcsdDfSnbPhjaNszlfgBBTSb%2FeW9IU%2FcjwwfSPahXibo3xafAIQa1D5LH%2BKMjI0sFeAEoCizQ%3D%3D HTTP/1.1"
   
6. Dec 2 10:12:26 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT js.aq.qq.com:443 HTTP/1.1"
   
7. Dec 2 10:12:26 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT res.wx.qq.com:443 HTTP/1.1"
   
8. Dec 2 10:12:26 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://bbs.chinaunix.net/ HTTP/1.1"
   
9. Dec 2 10:12:27 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://bbs.chinaunix.net/data/cache/style_3_common.css?HLC HTTP/1.1"
   
10. Dec 2 10:12:27 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://bbs.chinaunix.net/data/cache/style_3_forum_index.css?HLC HTTP/1.1"
    
11. Dec 2 10:12:27 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://bbs.chinaunix.net/static/js/common.js?HLC HTTP/1.1"
    
12. Dec 2 10:12:28 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT res.wx.qq.com:443 HTTP/1.1"
    
13. Dec 2 10:20:16 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT im.mukewang.com:80 HTTP/1.1"
    
14. Dec 2 10:20:16 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://coding.imooc.com/static/lib/login-regist/tpl/erweima.js?v=201611280061 HTTP/1.1"
    
15. Dec 2 10:20:16 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://szimg.mukewang.com/57ea56bd0001f6cb01400140-200-200.jpg HTTP/1.1"
    
16. Dec 2 10:20:17 ooxx-term[2060]: ooxx_pool/s1 200 {trustd (unknown version) CFNetwork/807.1.3 Darwin/16.1.0 (x86_64)} "GET http://ocsp.comodoca.com/MFYwVKADAgEAME0wSzBJMAkGBSsOAwIaBQAEFHrhPuigxCostCjL56YFRhlA4qHpBBSQr2o6lFoL2JDqElZz30O0Oija5wIQVY0%2F9sceAGHVl7iNuX8gzA%3D%3D HTTP/1.1"
    
17. Dec 2 10:20:53 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "POST http://python.jobbole.com/wp-admin/admin-ajax.php HTTP/1.1"
    
18. Dec 2 10:21:31 ooxx-term[2060]: ooxx_pool/s1 200 {trustd (unknown version) CFNetwork/807.1.3 Darwin/16.1.0 (x86_64)} "GET http://g.symcd.com/MEkwR6ADAgEAMEAwPjA8MAkGBSsOAwIaBQAEFLG0OReQFreXeVAR8WC51KI82%2B3uBBQA%2BSrDQZG2ycK4PlXywJcRE6AHIAIDAjp3 HTTP/1.1"
    
19. Dec 2 10:12:31 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://cu.img168.net/static/image/common/tubiao/common_283_icon.png HTTP/1.1"
    
20. Dec 2 10:12:31 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://cu.img168.net/static/image/common//logo.gif HTTP/1.1"
    
21. Dec 2 10:12:31 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://cu.img168.net/static/image/common/tubiao/common_233_icon.png HTTP/1.1"
    
22. Dec 2 10:12:31 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://bbs.chinaunix.net/data/attachment/common/icon/clearcase.gif HTTP/1.1"
    
23. Dec 2 10:14:00 ooxx-term[2060]: ooxx_pool/s1 200 {trustd (unknown version) CFNetwork/807.1.3 Darwin/16.1.0 (x86_64)} "GET http://ocsp.int-x3.letsencrypt.org/MFgwVqADAgEAME8wTTBLMAkGBSsOAwIaBQAEFH7maudymrP8%2BKIgZGwWoS1gcQhdBBSoSmpjBH3duubRObemRWXv86jsoQISA5qXFMT5ISpagmnHsXfXtpIF HTTP/1.1"
    
24. Dec 2 10:16:56 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET http://www.wooyun.org/favicon.ico HTTP/1.1"
    
25. Dec 2 10:17:46 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT dn-linuxcn.qbox.me:443 HTTP/1.1"
    
26. Dec 2 10:19:05 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "CONNECT www.bell.ca:443 HTTP/1.1"
    
27. Dec 2 10:19:15 ooxx-term[2060]: ooxx_pool/s1 200 {trustd (unknown version) CFNetwork/807.1.3 Darwin/16.1.0 (x86_64)} "GET http://support.typora.io//MEcwRaADAgEAMD4wPDA6MAkGBSsOAwIaBQAEFB0jb44GS5cf3d%2BhFSPhruUCvisbBBQ6moUHEGcotu%2F2vQVBbiDBlNoP3gIBBw%3D%3D HTTP/1.1"
    
28. Dec 2 10:23:53 ooxx-term[2060]: ooxx_pool/s1 200 {Mozilla/5.0 (Macintosh; Int el Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36} "GET https://www.iperf.fr/recv/gs.gif?gsver=3.4.0.4&gscmd=hb&gssrvid=GWD-000673&gsuid=80 6453484s4rbw17&gssid=80645348ochre617&pvid=806453487gf4rf17&gsltime=1480674232289&gstmzone=8&rd =4r45p&pld=84&gsst=0&gswh=734 HTTP/1.1"

复制代码

如下是我的输出结果：

1. cat cu.sh
   
2. #!/bin/sh
   
3. #
   
4. cat cu.txt |sort |uniq|egrep -i 'get|post|connect'|cut -d'"' -f2|cut -d' ' -f2|sort|uniq>out.txt
   
5. 
   
6. cat out.txt|grep -i '^http'>http.txt
   
7. cat out.txt|grep -iv '^http'>nohttp.txt
   
8. 
   
9. cat http.txt|cut -d'/' -f3|sort|uniq|awk -F'.' '{print "."$(NF-1)"."$NF}'|sort|uniq>out1.txt
   
10. cat nohttp.txt|cut -d':' -f1|awk -F'.' '{print "."$(NF-1)"."$NF}'>out2.txt
    
11. 
    
12. cat out1.txt out2.txt >ooxx.txt
    
13. cat ooxx.txt|sort|uniq>result.txt
    
14. rm ./out.txt
    
15. rm ./http.txt
    
16. rm ./nohttp.txt
    
17. rm ./out1.txt
    
18. rm ./out2.txt
    
19. rm ./ooxx.txt

复制代码

1. sh cu.sh
   
2. 
   
3. wc -l result.txt
   
4. 16 result.txt
   
5. 
   
6. more result.txt
   
7. .bell.ca
   
8. .chinaunix.net
   
9. .comodoca.com
   
10. .icloud.com
    
11. .img168.net
    
12. .imooc.com
    
13. .iperf.fr
    
14. .ipip.net
    
15. .jobbole.com
    
16. .letsencrypt.org
    
17. .mukewang.com
    
18. .qbox.me
    
19. .qq.com
    
20. .symcd.com
    
21. .typora.io
    
22. .wooyun.org

复制代码

回复 13# yinyuemi

1. awk 'match($0,/.*"(GET|POST|CONNECT) ([^ ]+)/,a){match(a[2],/.*(\.[^.]+\.(net|com))/,b);if(!c[b[1]]++)print b[1]}' cu.txt
   
2. .icloud.com
   
3. .ipip.net
   
4. .qq.com
   
5. .symcd.com
   
6. .chinaunix.net
   
7. .mukewang.com
   
8. .imooc.com
   
9. .comodoca.com
   
10. .jobbole.com
    
11. .img168.net
    

复制代码

1. awk 'match($0,/.*"(GET|POST|CONNECT) ([^ ]+)/,a){match(a[2],/.*(\.[^.]+\.(net|com))/,b);if(!c[b[1]]++)print b[1]}' cu.txt|wc -l
   
2. 11

复制代码

回复 5# moperyblue

1. awk '$(NF-2)~/get|post|connect/{match($(NF-1),/(http:\/\/)?[^/]*((\.[^/]*){2})[/:].*/,a)}!b[a[2]]++{print a[2]}' IGNORECASE=1 cu.txt
   
2. .icloud.com
   
3. .ipip.net
   
4. .qq.com
   
5. .symcd.com
   
6. .chinaunix.net
   
7. .mukewang.com
   
8. .imooc.com
   
9. .comodoca.com
   
10. .jobbole.com
    
11. .img168.net
    
12. .letsencrypt.org
    
13. .wooyun.org
    
14. .qbox.me
    
15. .bell.ca
    
16. .typora.io

复制代码

1. awk '$(NF-2)~/get|post|connect/{match($(NF-1),/(http:\/\/)?[^/]*((\.[^/]*){2})[/:].*/,a)}!b[a[2]]++{print a[2]}' IGNORECASE=1 cu.txt|wc -l
   
2. 15

复制代码

回复 6# sunzhiguolu

1. perl -ne '{$h{"$1\n"}++ if(/((?>\.\w+){2})(?=:|\/)/)}END{print keys %h}' cu.txt
   
2. .qbox.me
   
3. .mukewang.com
   
4. .typora.io
   
5. .iperf.fr
   
6. .ipip.net
   
7. .symcd.com
   
8. .wooyun.org
   
9. .imooc.com
   
10. .jobbole.com
    
11. .img168.net
    
12. .qq.com
    
13. .bell.ca
    
14. .comodoca.com
    
15. .letsencrypt.org
    
16. .icloud.com
    
17. .chinaunix.net

复制代码

1. perl -ne '{$h{"$1\n"}++ if(/((?>\.\w+){2})(?=:|\/)/)}END{print keys %h}' cu.txt|wc -l
   
2. 16
   
3. 
   
4. 您的结果准确无误，唯在下不才，用不起perl而已，非是对语言有偏见，此事需讲明。对仁兄的宽广胸襟，在下深感敬佩，如有失礼之处，忘您勿怪责个。
   

复制代码

sunzhiguolu

回复 21# 方兆国儿

yinyuemi

方兆国儿 发表于 2016-12-05 22:11
回复 6# sunzhiguolu

亦要感谢您的回复。测试结果显示，数据存在缺漏（与我计算的相差90多行）

为了awk，手足都放弃了