fastdfs 4.08版本，appender疯狂发送后会报错

xcl3721

同步能够就行进行但是很多错误输出

系统环境信息：
uname -a
Linux test12.com 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
centos 5.2
------------------------
[root@test12 fastdfstest]# rpm -qa libevent
libevent-1.4.13-1
-----------------------
[root@test12 fastdfstest]# ldd /usr/local/bin/fdfs_storaged
        linux-vdso.so.1 =>  (0x00007fff59782000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000033ba800000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000033ba000000)
        libevent-1.4.so.2 => /usr/lib64/libevent-1.4.so.2 (0x00002b3e04357000)
        libc.so.6 => /lib64/libc.so.6 (0x00000033b9c00000)
        /lib64/ld-linux-x86-64.so.2 (0x00000033b9800000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00000033bd000000)
        librt.so.1 => /lib64/librt.so.1 (0x00000033bb000000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00000033c0000000)
------------------------------------------
[root@test15 FastDFS4.0.8]# pstack  32411
Thread 10 (Thread 0x41735940 (LWP 32412)):
#0  0x0000003df50d44b8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b580ec962c0 in ?? () from /usr/lib64/libevent-1.4.so.2
#2  0x00002b580ec894d1 in event_base_loop () from /usr/lib64/libevent-1.4.so.2
#3  0x000000000042f220 in work_thread_entrance ()
#4  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x417b6940 (LWP 32413)):
#0  0x0000003df50d44b8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b580ec962c0 in ?? () from /usr/lib64/libevent-1.4.so.2
#2  0x00002b580ec894d1 in event_base_loop () from /usr/lib64/libevent-1.4.so.2
#3  0x000000000042f220 in work_thread_entrance ()
#4  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x41855940 (LWP 32414)):
#0  0x0000003df5c0e047 in fsync () from /lib64/libpthread.so.0
#1  0x0000000000411ed6 in log_fsync ()
#2  0x00000000004121de in doLogEx ()
#3  0x00000000004123cb in doLog ()
#4  0x00000000004124ba in logDebug ()
#5  0x0000000000426275 in storage_sync_append_file ()
#6  0x000000000042e49e in storage_deal_task ()
#7  0x0000000000434b8d in client_sock_read ()
#8  0x00002b580ec8959a in event_base_loop () from /usr/lib64/libevent-1.4.so.2
#9  0x000000000042f220 in work_thread_entrance ()
#10 0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#11 0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x418d6940 (LWP 32415)):
#0  0x0000003df50d44b8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b580ec962c0 in ?? () from /usr/lib64/libevent-1.4.so.2
#2  0x00002b580ec894d1 in event_base_loop () from /usr/lib64/libevent-1.4.so.2
#3  0x000000000042f220 in work_thread_entrance ()
#4  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x41957940 (LWP 32416)):
#0  0x0000003df509a541 in nanosleep () from /lib64/libc.so.6
#1  0x0000003df509a364 in sleep () from /lib64/libc.so.6
#2  0x000000000041f9d1 in tracker_report_thread_entrance ()
#3  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x419d8940 (LWP 32417)):
#0  0x0000003df509a541 in nanosleep () from /lib64/libc.so.6
#1  0x0000003df509a364 in sleep () from /lib64/libc.so.6
#2  0x00000000004166a6 in sched_thread_entrance ()
#3  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x41a59940 (LWP 3241):
#0  0x0000003df5c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004362b8 in dio_thread_entrance ()
#2  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#3  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x41ada940 (LWP 32419)):
#0  0x0000003df5c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004362b8 in dio_thread_entrance ()
#2  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#3  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x41b5b940 (LWP 32420)):
#0  0x0000003df509a541 in nanosleep () from /lib64/libc.so.6
#1  0x0000003df50cdb24 in usleep () from /lib64/libc.so.6
#2  0x0000000000432e5a in storage_sync_thread_entrance ()
#3  0x0000003df5c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003df50d40cd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b580ee9f840 (LWP 32411)):
#0  0x0000003df5c0db3b in accept () from /lib64/libpthread.so.0
#1  0x000000000042eb4c in accept_thread_entrance ()
#2  0x000000000042edf9 in storage_accept_loop ()
#3  0x000000000040c1dc in main ()

------------------------
[root@test15 FastDFS4.0.8]# strace -p 32411
Process 32411 attached - interrupt to quit
accept(5,




<unfinished ...>
Process 32411 detached

-----------------------------------------------------------------------------------
错误信息：开了debug，fdfs都是使用的4.08默认配置

2014-08-08 17:03:47] ERROR - file: tracker_proto.c, line: 48, server: 172.1.4.16:23000, response status 17 != 0
[2014-08-08 17:03:47] DEBUG - file: storage_service.c, line: 5933, client ip: 172.1.4.16, file /home/yuqing/fastdfs/data/00/00/rAEEEFPi7pCEFoiXAAAAAG3nojE601.txt,  my file size: 3852779 >= src file size: 3813513, do not append
[2014-08-08 17:03:47] ERROR - file: tracker_proto.c, line: 48, server: 172.1.4.16:23000, response status 17 != 0
[2014-08-08 17:03:47] DEBUG - file: storage_service.c, line: 5933, client ip: 172.1.4.16, file /home/yuqing/fastdfs/data/00/00/rAEEEFPi7pCEFoiXAAAAAG3nojE601.txt,  my file size: 3852779 >= src file size: 3813520, do not append
[2014-08-08 17:03:47] ERROR - file: tracker_proto.c, line: 48, server: 172.1.4.16:23000, response status 17 != 0
[2014-08-08 17:03:47] DEBUG - file: storage_service.c, line: 5933, client ip: 172.1.4.16, file /home/yuqing/fastdfs/data/00/00/rAEEEFPi7pCEFoiXAAAAAG3nojE601.txt,  my file size: 3852779 >= src file size: 3813527, do not append
[2014-08-08 17:03:47] ERROR - file: tracker_proto.c, line: 48, server: 172.1.4.16:23000, response status 17 != 0
[2014-08-08 17:03:47] DEBUG - file: storage_service.c, line: 5933, client ip: 172.1.4.16, file /home/yuqing/fastdfs/data/00/00/rAEEEFPi7pCEFoiXAAAAAG3nojE601.txt,  my file size: 3852779 >= src file size: 3813534, do not append


-----------------------------------------------------------
以下为服务器状态
You have new mail in /var/spool/mail/root
[root@test12 fastdfstest]# fdfs_monitor /etc/fdfs/client.conf
[2014-08-08 16:53:39] DEBUG - base_path=/home/yuqing/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 172.1.4.16:22122

group count: 1

Group 1:
group name = group1
disk total space = 535016 MB
disk free space = 290239 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 172.1.4.15
                ip_addr = 172.1.4.15 (172-1-4-15.lightspeed.hstntx.sbcglobal.net)  ACTIVE
                http domain =
                version = 4.08
                join time = 2014-08-07 10:57:30
                up time = 2014-08-08 16:48:39
                total storage = 535016 MB
                free storage = 290239 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id= 172.1.4.16
                if_trunk_server= 0
                total_upload_count = 257
                success_upload_count = 257
                total_append_count = 123262
                success_append_count = 123262
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 749360077
                success_upload_bytes = 749360077
                total_append_bytes = 791829
                success_append_bytes = 791829
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 965425960
                success_sync_in_bytes = 953855414
                total_sync_out_bytes = 753804924
                success_sync_out_bytes = 753088027
                total_file_open_count = 124571
                success_file_open_count = 124571
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 130722
                success_file_write_count = 130722
                last_heart_beat_time = 2014-08-08 16:58:59
                last_source_update = 2014-08-08 16:47:53
                last_sync_update = 2014-08-08 16:59:40
                last_synced_timestamp = 2014-08-08 16:47:03 (16s delay)
        Storage 2:
                id = 172.1.4.16
                ip_addr = 172.1.4.16 (sphinx_host_v2)  ACTIVE
                http domain =
                version = 4.08
                join time = 2014-08-07 10:58:57
                up time = 2014-08-07 10:58:57
                total storage = 904564 MB
                free storage = 793465 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id=
                if_trunk_server= 0
                total_upload_count = 322
                success_upload_count = 322
                total_append_count = 140032
                success_append_count = 140032
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 14
                success_download_count = 14
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 950407622
                success_upload_bytes = 950407622
                total_append_bytes = 902418
                success_append_bytes = 902418
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 7404938
                success_download_bytes = 7404938
                total_sync_in_bytes = 764762045
                success_sync_in_bytes = 753088027
                total_sync_out_bytes = 954566827
                success_sync_out_bytes = 953855414
                total_file_open_count = 144751
                success_file_open_count = 144751
                total_file_read_count = 42
                success_file_read_count = 42
                total_file_write_count = 150893
                success_file_write_count = 150893
                last_heart_beat_time = 2014-08-08 16:59:07
                last_source_update = 2014-08-08 16:47:19
                last_sync_update = 2014-08-08 16:59:07
                last_synced_timestamp = 2014-08-08 16:47:51 (2s delay)




----------------------------------------------------------------------------------
以下内容是我做压测的代码，跑几次必现

<?php

$starttime = time();
$totalcount = 0;
$success_count = 0;
$del_count = 0;

while(true)
{
        $totalcount++;
        $info = FastFile::append_file_content($totalcount."\r\n","group1","M00/00/00/rAEEEFPi7pCEFoiXAAAAAG3nojE601.txt";
        if (!$info) {
            var_dump(FastFile::getError());
        }else{
                $success_count ++;
                //$res = FastFile::delete_file($info["group_name"], $info["filename"]);
                //if($res)
                //{
                //      $del_count ++;
                //}
        }
        if($totalcount % 100==0)
        {
                echo "cost:". (time() - $starttime);
                echo " total:".$totalcount." success:".$success_count." del:".$del_count." \r\n";
        }
        if($totalcount >= 20000)break;
}
echo "--------------------------------------------------------\r\n";
echo "cost:". (time() - $starttime);
echo " total:".$totalcount." success:".$success_count." \r\n";


    /**
     * 初始化，系统自动调用，如果想切换链接的storage也可以使用这个命令
     * @return bool
     */
    public function init()
    {
        //获取资源查找器的链接信息
        self:tracker = fastdfs_tracker_get_connection();
        if (!fastdfs_active_test(self:tracker)) {
//----------------------------------------------------------------------------------还有这里会失败~~~~~~~~~~~~~~~~test测试会失败~~~~~~~~~~~~~~~~~当这个错误出现的时候
            return false;
        }   
        
        //获取存储服务器
        self:storage = fastdfs_tracker_query_storage_store();
        if (!self:storage) {
            return false;
        }
        return true;
    }

  public static function append_file_content($content, $group_name, $remote_filename)
    {
        if (!self:tracker || !self:storage) {
            if (!self::init()) {
                return false;
            }   
        }   
        return fastdfs_storage_append_by_filebuff($content, $group_name, $remote_filename, self:tracker, self:storage);
    }