- 论坛徽章:
- 0
|
本帖最后由 gg9654 于 2015-01-28 17:17 编辑
最近刚好用到FastDFS的python客户端,发现会有接收数据缺失的现象,并修复了一个socket接收代码的一个bug,如下
老代码(fdfs_client/connection.py):- def tcp_recv_response(conn, bytes_size, buffer_size = 4096):
- '''Receive response from server.
- It is not include tracker header.
- arguments:
- @conn: connection
- @bytes_size: int, will be received byte_stream size
- @buffer_size: int, receive buffer size
- @Return: tuple,(response, received_size)
- '''
- response = ''
- total_size = 0
- try:
- while 1:
- if bytes_size <= buffer_size:
- resp = conn._sock.recv(buffer_size)
- response +=resp
- total_size += len(resp)
- break
- resp = conn._sock.recv(buffer_size)
- response += resp
- total_size += len(resp)
- bytes_size = bytes_size - buffer_size #注意这个地方的有问题
- except (socket.error, socket.timeout), e:
- raise ConnectionError('[-] Error: while reading from socket: (%s)' \
- % e.args)
- return (response, total_size)
复制代码 新代码:- def tcp_recv_response(conn, bytes_size, buffer_size = 4096):
- '''Receive response from server.
- It is not include tracker header.
- arguments:
- @conn: connection
- @bytes_size: int, will be received byte_stream size
- @buffer_size: int, receive buffer size
- @Return: tuple,(response, received_size)
- '''
- response = ''
- total_size = 0
- try:
- while 1:
- if bytes_size <= buffer_size:
- resp = conn._sock.recv(buffer_size)
- response +=resp
- total_size += len(resp)
- break
- resp = conn._sock.recv(buffer_size)
- response += resp
- recv_size = len(resp)
- total_size += recv_size
- bytes_size = bytes_size - recv_size
- except (socket.error, socket.timeout), e:
- raise ConnectionError('[-] Error: while reading from socket: (%s)' \
- % e.args)
- return (response, total_size)
复制代码 修改说明:- bytes_size = bytes_size - buffer_size
复制代码 改为- recv_size = len(resp)
- total_size += recv_size
- bytes_size = bytes_size - recv_size
复制代码 在socket的接收的时候,bytes_size 应该是减去实际接收到数据的大小(即recv_size = len(resp)),而不是减去固定的buff_size, 这样很容易造成接收不完整,从而导致数据缺失
在附件中有我的修改后的版本,主要的修改地方有两个:
1、如上
2、在storage_client.py新增加一个tcp_recv_buff,用于收取buff数据- def tcp_recv_buff(conn, file_size, buffer_size=4096):
- '''
- Receive buff from server, fragmented it while receiving and write to disk.
- arguments:
- @conn: connection
- @file_size: int, remote file size
- @buffer_size: int, receive buffer size
- @Return int: file size if success else raise ConnectionError.
- '''
- total_file_size = 0
- flush_size = 0
- remain_bytes = file_size
- respone = ""
- print remain_bytes
- while remain_bytes > 0:
- try:
- if remain_bytes >= buffer_size:
- file_buffer, recv_size = tcp_recv_response(conn, buffer_size, \
- buffer_size)
- else:
- file_buffer, recv_size = tcp_recv_response(conn, remain_bytes, \
- buffer_size)
- print recv_size
- respone += file_buffer
- remain_bytes -= recv_size
- total_file_size += recv_size
- except ConnectionError, e:
- raise ConnectionError('[-] Error: while downloading buff(%s).' % e.args)
- except IOError, e:
- raise DataError('[-] Error: while writting local buff(%s).' % e.args)
- return (respone,total_file_size)
复制代码 3、在storage_client.py中修改_storage_do_download_file里FDFS_DOWNLOAD_TO_BUFFER的调用函数- recv_buffer, total_recv_size = tcp_recv_buff(store_conn, th.pkg_len)
复制代码
fdfs_client-py-1.2.6_ex.zip
(193.28 KB, 下载次数: 15)
备注:这个python客户端问题比较多,效率不高,自己封装了一个,底层采用FastDFS C++接口,有兴趣的朋友可以看看,源码地址:github.com/cosysun/FastDFSClient_Python.git
|
|