FastDFS storage启动报错，读取binlog文件失败的异常问题，请余大及各位指点

mangc

今天启动storage时遇到这样的问题，下面是启动时的storaged.log 日志内容：

[2014-06-20 22:09:10] INFO - FastDFS v4.05, base_path=/fddata/pan/fastdfs, store_path_count=1, subdir_count_per_path=32, group_name=group6, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2014-06-20 22:09:10] INFO - file: storage_param_getter.c, line: 187, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=1, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, store_slave_file_use_link=0
[2014-06-20 22:09:10] INFO - file: storage_func.c, line: 174, tracker_client_ip: 192.168.124.72, my_server_id_str: 192.168.124.72, g_server_id_in_filename: 1216129216
[2014-06-20 22:09:10] INFO - local_host_ip_count: 2,  127.0.0.1  192.168.124.72
[2014-06-20 22:09:10] INFO - file: tracker_client_thread.c, line: 308, successfully connect to tracker server 192.168.147.123:22122, as a tracker client, my ip is 192.168.124.72
[2014-06-20 22:09:10] INFO - file: tracker_client_thread.c, line: 1124, tracker server 192.168.147.123:22122, set tracker leader: 192.168.147.123:22122
[2014-06-20 22:09:10] INFO - file: tracker_client_thread.c, line: 1185, I am the the trunk server 192.168.124.72:23000
[2014-06-20 22:09:10] ERROR - file: trunk_mgr/trunk_sync.c, line: 837, read data from binlog file "/fddata/pan/fastdfs/data/trunk/binlog" fail, file offset: 27429432, read item count: 1 < 8
[2014-06-20 22:09:10] ERROR - file: trunk_mgr/trunk_sync.c, line: 779, read from binlog file "/fddata/pan/fastdfs/data/trunk/binlog" fail, file offset: 27429433, line buffer size: 128 is too small! <= line length: 550
[2014-06-20 22:09:10] ERROR - file: trunk_mgr/trunk_mem.c, line: 777, trunk load fail, errno: 28, error info: No space left on device
[2014-06-20 22:09:11] INFO - file: tracker_client_thread.c, line: 308, successfully connect to tracker server 192.168.147.122:22122, as a tracker client, my ip is 192.168.124.72
[2014-06-20 22:09:11] INFO - file: trunk_mgr/trunk_sync.c, line: 1068, trunk sync thread to storage server 192.168.124.73:23000 started
[2014-06-20 22:09:11] INFO - file: tracker_client_thread.c, line: 308, successfully connect to tracker server 192.168.147.123:22122, as a tracker client, my ip is 192.168.124.72
[2014-06-20 22:09:11] ERROR - file: trunk_mgr/trunk_sync.c, line: 1141, connect to storage server 192.168.124.73:23000 fail, errno: 111, error info: Connection refused
[2014-06-20 22:09:11] INFO - file: tracker_client_thread.c, line: 1124, tracker server 192.168.147.123:22122, set tracker leader: 192.168.147.123:22122
[2014-06-20 22:09:11] WARNING - file: tracker_client_thread.c, line: 1179, I am already the trunk server 192.168.124.72:23000, may be the tracker server restart
[2014-06-20 22:09:17] ERROR - file: trunk_mgr/trunk_mem.c, line: 1378, I am not inited!
[2014-06-20 22:09:17] ERROR - file: storage_nio.c, line: 314, client ip: 127.0.0.1, pkg length: -4431937177564820894 < 0
[2014-06-20 22:09:22] ERROR - file: trunk_mgr/trunk_mem.c, line: 1378, I am not inited!
[2014-06-20 22:09:31] ERROR - file: trunk_mgr/trunk_mem.c, line: 1378, I am not inited!
[2014-06-20 22:09:31] ERROR - file: trunk_mgr/trunk_mem.c, line: 1378, I am not inited!

期初看到 trunk load fail, errno: 28, error info: No space left on device 以为是磁盘空间的问题
分别用df查看了磁盘空间和inode节点使用情况，都是正常的，没有到不足的情况

后来又查看了下日志中报错读取失败的 binlog文件(UTF8打开)，发现和其他正常的storage server不同，除了有一些记录时间，文件等信息的内容，多出很多乱码的内容，如下

1401804499 D 0 0 22 871 48095760 1051468
1401804501 D 0 0 22 871 49147228 17961636
1401804501 A 0 0 22 871 55574092 11534772
1401804501 A 0 0 22 871 49147228 6426864
1401804501 D 0 0 22 871 49147228 6426864
1401804505 D 0 0 22 869 67107883 981
1401804505 A 0 0 22 869 67107995 869
1401804505 A 0 0 22 869 67107883 112
1401804505 D 0 0 22 869 67107883 112
�mbZn�Ȉ���t�n�v��-���Q�U�屶�\��j�0K7Z3o�Ah���6���e        ���΄S�Z���N��2��
J��53��uML⒖�srX���?��Y籶e���IY�1pn8g�8'�sL:���*ʉ�X���CLf����y��_'%�l�ӖRB
�ת:V=V�R69��+%�.QXֺa��Ԉ�Na�?��[�i��K���7�gq?o��5bx���0���w�:c�]��c����U��##~��,�gQ�8}b�m��0��~�zj��hzSv��t8�?X=���4�
��:�K%��7z��6á���+^�}�lb��(��{��i��f��(\�ǣ`,�M�t�������k:�yaXEd�p�.��9�a��cm��

不知道是不是binlog文件损坏导致启动时不能读取的，还是应该通过其他什么思路去进一步定位？

另外，如果是binlog文件损坏了，这个文件能不能重建，因为之前看到的资料说，这个binlog记录的是文件的操作记录，如果删除，组内新增节点会不能同步已有文件

不知道有没有其他人遇到过类似情况。。。

请余大指导

半岛北边

都没人帮忙唉 lz binlog能重建吗

happy_fish100

文件内容错乱了

zhouc_yan

楼组后来又解决吗