关于搜索关键词的问题

jidi_78

"http://www.baidu.com/baidu?word=%D2%B6%CB%E1%D3%A6%D4%DA%BB%B3%D4%D0%C7%B0%B6%E0%B3%A4%CA%B1%BC%E4%B3%D4&select=1&tn=zhangxia019_pg"**/ask/detail/20790
"http://www.baidu.com/s?lm=0&si=&rn=10&tn=webto_cb&ie=gb2312&ct=0&wd=%BB%B3%D4%D0%C7%B0%BF%C9%D2%D4%B3%D4%D2%A9%C2%F0&pn=30&ver=0&cl=3&uim=0&usm=0&f=1"**/ask/detail/93521

现在知道word=后%D2%B6%CB%E1%D3%A6%D4%DA%BB%B3%D4%D0%C7%B0%B6%E0%B3%A4%CA%B1%BC%E4%B3%D4，或wd=后的%BB%B3%D4%D0%C7%B0%BF%C9%D2%D4%B3%D4%D2%A9%C2%F0的是用户从百度搜索的关键词，请问怎样把这些字符转换成文字？

另外google的又是怎么做的呢？请教大家？

starwing83

def get_str(s):
    li = map(lambda x: int(x, 16), filter(bool, s.split('%')))
    return ''.join(map(chr, li)).decode('gbk')

print get_str('%D2%B6%CB%E1%D3%A6%D4%DA%BB%B3%D4%D0%C7%B0%B6%E0%B3%A4%CA%B1%BC%E4%B3%D4')
print get_str('%BB%B3%D4%D0%C7%B0%BF%C9%D2%D4%B3%D4%D2%A9%C2%F0')

Shell_HAT

URL解码：
http://bbs.chinaunix.net/thread-934872-1-1.html
http://bbs.chinaunix.net/thread-982743-1-1.html
http://bbs.chinaunix.net/thread-1117156-1-1.html
http://bbs.chinaunix.net/thread-1648997-1-1.html

jidi_78

谢谢帮忙。
用 printf "${line//\%/\\x}\n"做的转换，转换出的结果大部分都没有问题，但有一部分的结果因为原日志中是这样的格式：\xd0\xc2\xb0\xe6\xc8\xfd\xd6\xbb\xd0\xdc
而不能转换，而且在转换的结果中有一小部分乱码的情况。

请教了其他高人，说日志中的编码有utf-8，gbk,gb2312几种格式....所以会有乱码和上面的字符格式


请教，上面的\xd0\xc2\xb0\xe6\xc8\xfd\xd6\xbb\xd0\xdc属于那种编码？在shell能根据日志的内容区分不同的编码吗？

jidi_78

1. "http://www.baidu.com/s?wd=%CA%D6%B3%AD%B1%A8%CD%BC%BB%AD&rsp=1&oq=%CD%BC%BB%AD&f=1&tn=site888_pg"**/user/showuser.php?uid=u104668001&tab=journal&journalid=986514&view=single
   
2. "http://www.baidu.com/s?wd=%B2%B8%C8%E9%C6%DA%B5%C4%B8%BE%C5%AE%C4%DC%B7%F1%CF%B4%BD%C5&tn=lqowen_1_pg"**/ask/myqa__view~qdetail,qid~132174
   
3. "http://www.baidu.com/s?tn=utf8kb_dg&wd=%C3%C0%C3%C0%BB%A8%BB%A8%B1%A6%B1%A6"**/user/showuser.php?uid=u13555199&view=sh_mv
   
4. "http://www.baidu.com/s?tn=sitehao123&bs=%D4%D0%B8%BE%C3%CE%BC%FB%B4%F3%CA%F7&f=3&wd=%BB%B3%C4%D0%BA%A2%D7%D3%B5%C4%D6%A2%D7%B4&oq=%BB%B3%C4%D0%BA%A2%D7%D3%B5%C4%D6%A2&rsp=0"**/forum/topic_599631
   
5. "http://www.baidu.com/s?bs=%C8%FD%B8%F6%D4%C2%B5%C4%B1%A6%B1%A6%BF%C9%D2%D4%B3%D4%CD%FA%D7%D0%D0%A1%C2%F8%CD%B7%C2%F0&f=3&wd=%C8%FD%B8%F6%D4%C2%B5%C4%B1%A6%B1%A6%BF%C9%D2%D4%B3%D4%CF%E3%BD%B6%C2%F0&oq=%C8%FD%B8%F6%D4%C2%B5%C4%B1%A6%B1%A6%BF%C9%D2%D4%B3%D4&rsp=4"**/ask/myqa__view~qdetail,qid~62885
   
6. "http://www.baidu.com/baidu?tn=lin7163862_dg&wd=%CD%C8%C9%CF%B3%A4%B8%F6%CB%AE%C5%DD"**/user/showuser.php?uid=u5362163547&tab=journal&journalid=2046435&view=single
   
7. "http://www.baidu.com/s?wd=%B1%A6%B1%A6%CB%B5%BB%B0%B2%BB%C7%E5&oq=%B1%A6%B1%A6%CB%B5%BB%B0&f=3&rsp=4"**/ask/myqa__view~qdetail,qid~53656
   
8. "http://www.baidu.com/baidu?word=%D0%A1%BA%A2%C1%F7%B1%C7%CC%E9%B3%D4%CA%B2%C3%B4%D2%A9&tn=kzxf_pg"**/ask/detail/145362
   
9. "http://www.baidu.com/s?wd=%D3%A4%B6%F9%B4%F3%B1%E3%D4%F6%B6%E0"**/ask/myqa__view~qdetail,qid~17309
   
10. "http://www.baidu.com/s?wd=%CD%AF%D7%B0%C5%FA%B7%A2%B3%C7&word=%CD%AF%D7%B0%C5%FA%B7%A2%B3%C7&tn=sitehao123"**/user/showuser.php?uid=u1463980896&tab=journal&view=single&journalid=1842389
    
11. "http://www.baidu.com/baidu?word=%C6%F0%C3%FB&tn=asp51_pg"**/topic/baobaoqiming
    
12. "http://www.baidu.com/s?newwindow=&tn=verycd_pg&wd=%B1%A6%B1%A6%D0%A1%BC%A6%D3%D0%BA%EC%B5%E3&btnG.x=38&btnG.y=21"**/ask/myqa__view~qdetail,qid~71963
    
13. "http://hap1.ucweb.com.cn:8040/baidu_groups/web/http/www.baidu.com/s?wd=%BD%F0%F7%EB%C6%F1%CA%C7%B3%D8%D6%D0%CE%EF&pn=6&tn=ucfly_8_dg&rn=6"**/forum/topic_376225
    
14. "http://www.baidu.com/s?wd=%C3%D5%D3%EF%B4%F3%C8%AB%BC%B0%B4%F0%B0%B8&oq=%C3%D5%D3%EF&f=3&rsp=1"**/user/showuser.php?uid=u226552265&tab=journal&view=single&journalid=1713889
    
15. "http://www.baidu.com/s?wd=%C4%D1%CD%FC%B4%F3%D1%A7%CB%EA%D4%C2&pn=10&ytf4shxBpp=aKqHWnRlVa"**/forum/topic.php?topicId=423105
    
16. "http://www.baidu.com/baidu?tn=5snow&word=%C8%F3%B7%CE%B5%C4%CB%AE%B9%FB"**/user/bookmarks.php?tag=%E6%B6%A6%E8%82%BA%E7%9A%84%E6%B0%B4%E6%9E%9C
    
17. "http://www.baidu.com/s?wd=%CA%AE%D2%BB%B8%F6%D4%C2%B1%A6%B1%A6%CA%B3%C6%D7&oq=shiyige&f=3&rsp=1"**/forum/topic_826773
    
18. "http://www.baidu.com/s?wd=%D0%A1%BA%A2%BF%C8%CB%D4%C6%AB%B7%BD&rsp=3&oq=%D0%A1%BA%A2%B7%B4%B8%B4%BF%C8%CB%D4&f=1&tn=sitehao123"**/user/showuser.php?uid=u143879991&tab=journal&journalid=2242132&view=single
    
19. "http://www.baidu.com/s?wd=%CD%AF%D7%B0%C5%FA%B7%A2%B3%C7&word=%CD%AF%D7%B0%C5%FA%B7%A2%B3%C7&tn=sitehao123"**/user/showuser.php?uid=u1463980896&tab=journal&view=single&journalid=1842389
    
20. "http://www.baidu.com/s?wd=%BF%CB%B7%FE%C0%A7%C4%D1%B5%C4%D3%CE%CF%B7"**/forum/topic_71008
    
21. "http://www.baidu.com/s?wd=11%B8%F6%D4%C2%B1%A6%B1%A6%C1%F7%B1%C7%D1%AA%D4%F5%C3%B4%B0%EC&tn=site888_pg&bar="**/ask/myqa__view~qdetail,qid~71473
    
22. "http://hap1.ucweb.com.cn:8040/baidu_groups/web/http/www.baidu.com/s?wd=%d0%a1%b6%f9%b2%bb%ba%c8%cb%ae%d4%f5%c3%b4%b0%ec&tn=ucfly_2_dg&rn=6"**/ask/myqa__view%7Eqdetail,qid%7E90465
    
23. "http://www.baidu.com/s?wd=%B3%D4%C1%CB%B1%DC%D4%D0%D2%A9%BA%F3%BB%B3%D4%D0%B6%D4%CC%A5%B6%F9%D3%D0%D3%B0%CF%EC%C2%F0"**/ask/myqa__view~qdetail,qid~19644
    
24. "http://www.baidu.com/s?wd=%D0%A1%BA%A2%CD%B7%C6%A4%B8%C9%D4%EF&word=%D0%A1%BA%A2%CD%B7%C6%A4%B8%C9%D4%EF&tn=sitehao123&oq=1&rsp=0&f=3"**/ask/myqa__view~qdetail,qid~48684
    
25. "http://www.baidu.com/s?wd=%C3%BF%CC%EC%B3%D4%B6%E0%C9%D9%BA%FA%C2%DC%B2%B7%BA%C3&pn=40&tn=baiducb6&f=1"**/user/showuser.php?uid=u108102527&tab=journal&journalid=306530&view=single
    
26. "http://www.baidu.com/s?wd=%C6%F0%B5%B6%C7%B9%B4%CC%CA%C7%D4%F5%C3%B4%BB%D8%CA%C2&word=%C6%F0%B5%B6%C7%B9%B4%CC%CA%C7%D4%F5%C3%B4%BB%D8%CA%C2&tn=sitehao123"**/ask/myqa__view~qdetail,qid~19514
    
27. "http://www.baidu.com/s?wd=%B6%CF%C4%CC%CA%B1%C4%CC%D5%CD%D4%F5%C3%B4%B0%EC&rsp=7&oq=%B1%A6%B1%A6%B6%CF%C4%CC%C1%BD%CC%EC%C1%CB%BB%B9%D6%D7%D5%CD%D4%F5%C3%B4%B0%EC&f=1&tn=site888_pg"**/ask/myqa__view~qdetail,qid~11167,pg~2
    
28. "http://www.baidu.com/s?wd=%B8%D0%C3%B0%C4%DC%B4%F2%BC%D7%C1%F7%D2%DF%C3%E7%C2%F0&pn=20&tn=sitehao123&f=1"**/ask/myqa__view~qdetail,qid~16074
    
29. "http://www.baidu.com/s?wd=domain%3Awww.gb256.com&pn=20"**/forum/topic_776810
    
30. "http://www.baidu.com/s?wd=%D7%C4%C4%BE%C4%F1%B6%F9%CD%AF%D3%F0%C8%DE%B7%FE&word=%D7%C4%C4%BE%C4%F1%B6%F9%CD%AF%D3%F0%C8%DE%B7%FE&tn=sitehao123"**/forum/topic_473921
    
31. "http://hap1.ucweb.com.cn:8040/baidu_groups/web/http/www.baidu.com/s?wd=%ba%c8%c4%cc%ba%f3%bf%c9%d2%d4%c1%a2%bc%b4%ba%c8%cb%ae%c2%f0&tn=ucfly_2_dg&rn=6"**/ask/myqa__view~qdetail,qid~25600
    
32. "http://www.baidu.com/s?wd=%BD%E4%D1%CC%D3%EB%D3%C5%C9%FA&word=%BD%E4%D1%CC%D3%EB%D3%C5%C9%FA&tn=sitehao123"**/know/knowdetail__kid~1065,ktype~article
    
33. "http://www.baidu.com/s?tn=youcome_pg&bs=%B0%B2%C8%B8%B5%B0+%BC%A6%B5%B0&f=8&wd=%F0%C6%F0%C8%B5%B0+%BC%A6%B5%B0"**/ask/myqa__view~qdetail,qid~6287,pg~2
    
34. "http://www.baidu.com/s?wd=%C8%D5%D2%B9%B5%DF%B5%B9%D4%F5%C3%B4%B0%EC&rsp=7&oq=%B1%A6%B1%A6%CB%AF%BE%F5%C8%D5%D2%B9%B5%DF%B5%B9&f=1&tn=dwso_5_dg"**/ask/myqa__view~qdetail,qid~70066
    
35. "http://www.baidu.com/s?wd=%C9%F2%D1%F4%BF%E1%C0%D6%B1%A6%B1%A6%C9%E3%D3%B0&pn=0"**/group/topic_410957
    
36. @
    
37. 
    
38.    

复制代码

上面是一小部分日志格式，供大家参考

rdcwayx

def get_str(s):
    li = map(lambda x: int(x, 16), filter(bool, s.split('%')))
    return ''.join( ...
starwing83 发表于 2010-05-15 10:19

sunceenjoy

用php吧。urldecode函数即可.
php文件编码为utf-8

starwing83

直接printf以后扔给iconv不行么？

XQKA

神奇。