使用beautfulSoup抓取网页数据并写入txt文件失败

maple412

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import urllib2
import re

link=''
path=r'E:/pydownload/1.txt'
f=open(path,'wb+')
for i in range(1,150):
    print "it is download the %d page:" % i
    new=link + str(i) + '.'+'shtml'
    req=urllib2.Request(new)
    fd=urllib2.urlopen(req)
    soup=BeautifulSoup(fd.read(),from_encoding="utf-8"
    ret=soup.find(id="content_1"
    for r in ret.stripped_strings:
        f.write(r)

f.close()
网页的内容是中文的。提示如下错误：
Traceback (most recent call last):
  File "E:\py_prj\test1.py", line 17, in <module>
    f.write(r)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(12

ma__

1. reload(sys)
   
2. sys.setdefaultencoding( "utf-8" )

复制代码

maple412

回复 2# ma__
没用，一样的效果

ma__

1. # -*- coding: utf-8 -*-
   
2. from bs4 import BeautifulSoup
   
3. import urllib2
   
4. import re,sys
   
5. reload(sys)
   
6. sys.setdefaultencoding( "utf-8" )
   
7. 
   
8. link='http:     //movie.douban.com/'
   
9. path=r'1.txt'
   
10. f=open(path,'wb+')
    
11. # print "it is download the %d page:" % i
    
12. # new=link + str(i) + '.'+'shtml'
    
13. req=urllib2.Request(link)
    
14. fd=urllib2.urlopen(req)
    
15. soup=BeautifulSoup(fd.read(),from_encoding="utf-8")
    
16. ret=soup.find(id="top-nav-appintro")
    
17. for r in ret.stripped_strings:
    
18.         f.write(r)
    
19.         print r
    
20. # f.close()
    

复制代码

To get rid of this warning, change this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "lxml"

豆瓣
3.0
和有趣的人做有趣的事
扫码直接下载
iPhone
·
Android
为什么我可以

ma__

r是unicode
那你写之前加行
r=r.encode('utf-8','ignore')
吧