Chinaunix

标题: 关于fastDFS中python客户端的接收数据不完整bug [打印本页]

作者: gg9654 时间: 2014-09-04 10:55
标题: 关于fastDFS中python客户端的接收数据不完整bug
本帖最后由 gg9654 于 2015-01-28 17:17 编辑

最近刚好用到FastDFS的python客户端，发现会有接收数据缺失的现象，并修复了一个socket接收代码的一个bug，如下

老代码（fdfs_client/connection.py）：

def tcp_recv_response(conn, bytes_size, buffer_size = 4096):
'''Receive response from server.
It is not include tracker header.
arguments:
@conn: connection
@bytes_size: int, will be received byte_stream size
@buffer_size: int, receive buffer size
@Return: tuple,(response, received_size)
'''
response = ''
total_size = 0
try:
while 1:
if bytes_size <= buffer_size:
resp = conn._sock.recv(buffer_size)
response +=resp
total_size += len(resp)
break
resp = conn._sock.recv(buffer_size)
response += resp
total_size += len(resp)
bytes_size = bytes_size - buffer_size #注意这个地方的有问题
except (socket.error, socket.timeout), e:
raise ConnectionError('[-] Error: while reading from socket: (%s)' \
% e.args)
return (response, total_size)

复制代码

新代码：

def tcp_recv_response(conn, bytes_size, buffer_size = 4096):
'''Receive response from server.
It is not include tracker header.
arguments:
@conn: connection
@bytes_size: int, will be received byte_stream size
@buffer_size: int, receive buffer size
@Return: tuple,(response, received_size)
'''
response = ''
total_size = 0
try:
while 1:
if bytes_size <= buffer_size:
resp = conn._sock.recv(buffer_size)
response +=resp
total_size += len(resp)
break
resp = conn._sock.recv(buffer_size)
response += resp
recv_size = len(resp)
total_size += recv_size
bytes_size = bytes_size - recv_size
except (socket.error, socket.timeout), e:
raise ConnectionError('[-] Error: while reading from socket: (%s)' \
% e.args)
return (response, total_size)

复制代码

修改说明：

bytes_size = bytes_size - buffer_size

复制代码

改为

recv_size = len(resp)
total_size += recv_size
bytes_size = bytes_size - recv_size

复制代码

在socket的接收的时候，bytes_size 应该是减去实际接收到数据的大小（即recv_size = len(resp)），而不是减去固定的buff_size, 这样很容易造成接收不完整，从而导致数据缺失

在附件中有我的修改后的版本，主要的修改地方有两个：

1、如上

2、在storage_client.py新增加一个tcp_recv_buff，用于收取buff数据

def tcp_recv_buff(conn, file_size, buffer_size=4096):
'''
Receive buff from server, fragmented it while receiving and write to disk.
arguments:
@conn: connection
@file_size: int, remote file size
@buffer_size: int, receive buffer size
@Return int: file size if success else raise ConnectionError.
'''
total_file_size = 0
flush_size = 0
remain_bytes = file_size
respone = ""
print remain_bytes
while remain_bytes > 0:
try:
if remain_bytes >= buffer_size:
file_buffer, recv_size = tcp_recv_response(conn, buffer_size, \
buffer_size)
else:
file_buffer, recv_size = tcp_recv_response(conn, remain_bytes, \
buffer_size)
print recv_size
respone += file_buffer
remain_bytes -= recv_size
total_file_size += recv_size
except ConnectionError, e:
raise ConnectionError('[-] Error: while downloading buff(%s).' % e.args)
except IOError, e:
raise DataError('[-] Error: while writting local buff(%s).' % e.args)
return (respone,total_file_size)

复制代码

3、在storage_client.py中修改_storage_do_download_file里FDFS_DOWNLOAD_TO_BUFFER的调用函数

recv_buffer, total_recv_size = tcp_recv_buff(store_conn, th.pkg_len)

复制代码

fdfs_client-py-1.2.6_ex.zip (193.28 KB, 下载次数: 15)

备注：这个python客户端问题比较多，效率不高，自己封装了一个，底层采用FastDFS C++接口，有兴趣的朋友可以看看，源码地址:github.com/cosysun/FastDFSClient_Python.git

作者: liuxuejin 时间: 2014-10-17 15:44
感谢，楼主有没有发现python版的客户端无法饮用sendfile的错误，如何解决呢？

作者: fl8866 时间: 2014-10-17 20:36
这个感觉如何https://github.com/hay86/fdfs_client-py回复 1# gg9654

作者: gg9654 时间: 2015-01-28 16:45
建议不要使用这个python版本，发现效率不高，最后自己封装了一个客户端，底层采用C++客户端，稍后会开源回复 2# liuxuejin

作者: liuxuejin 时间: 2015-03-10 16:50
回复 4# gg9654

那么靠谱期待你的最新客户端，有github么，可以尽早放出来让大家看看

欢迎光临 Chinaunix (http://bbs.chinaunix.net/)

Powered by Discuz! X3.2