FastDFS运行过程中出现服务器假死现象

yangchun98

FastDFS运行一段时间后（大约一个多月左右）出现服务器假死现象，问题发生时，ssh也无法连接到fastdfs服务器，必须手动重新服务器恢复。系统中配置的定时任务脚本也停止运行了，下面是我的storage服务器运行的整个log，期间也出现过几次假死现象，都是通过手工重启服务器恢复的。

[2012-01-06 13:09:54] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-01-06 13:10:19] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-01-06 13:10:21] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245
[2012-01-06 13:10:22] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-01-06 13:10:51] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.213:22122, set tracker leader: 192.168.0.213:22122
[2012-01-06 13:10:53] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-01-06 13:11:51] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-01-14 11:35:52] ERROR - file: tracker_client_thread.c, line: 985, tracker server 192.168.0.213:22122, recv data fail, errno: 110, error info: Connection timed out.
[2012-01-14 11:35:54] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.212:22122, set tracker leader: 192.168.0.212:22122
[2012-01-14 11:36:09] ERROR - file: tracker_client_thread.c, line: 271, connect to tracker server 192.168.0.213:22122 fail, errno: 111, error info: Connection refused
[2012-01-14 11:36:39] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, continuous fail count: 1, as a tracker client, my ip is 192.168.0.245
[2012-01-25 05:08:46] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-01-25 05:08:46] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-01-25 05:08:46] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245
[2012-01-25 05:08:47] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-01-25 05:08:47] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.212:22122, set tracker leader: 192.168.0.212:22122
[2012-01-25 05:09:16] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-03-03 17:49:53] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.213:22122, set tracker leader: 192.168.0.213:22122
[2012-03-03 17:49:54] ERROR - file: tracker_client_thread.c, line: 985, tracker server 192.168.0.212:22122, recv data fail, errno: 110, error info: Connection timed out.
[2012-03-03 17:50:12] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-04 04:04:12] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-03-13 07:45:25] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-03-13 07:45:25] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-03-13 07:45:26] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-13 07:45:26] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.213:22122, set tracker leader: 192.168.0.213:22122
[2012-03-13 07:45:27] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-13 07:45:27] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-03-13 07:48:49] ERROR - file: storage_service.c, line: 5447, call stat fail, logic file: M00/01/25/wKgAC09eGcHKOSO5AAKY9kuW1vQ3690620, error no: 2, error info: No such file or directory
[2012-03-13 08:08:44] CRIT - file: fdfs_storaged.c, line: 400, catch signal 15, program exiting...
[2012-03-13 08:08:45] INFO - exit nomally.
[2012-03-13 08:09:38] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-03-13 08:09:38] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB[2012-03-13 08:09:39] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245[2012-03-13 08:09:39] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.213:22122, set tracker leader: 192.168.0.213:22122[2012-03-13 08:09:39] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000[2012-03-13 08:09:40] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-13 08:36:11] CRIT - file: fdfs_storaged.c, line: 400, catch signal 15, program exiting...
[2012-03-13 08:38:04] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-03-13 08:38:04] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-03-13 08:38:04] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-03-13 08:38:04] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-03-13 08:38:04] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-13 08:38:04] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.213:22122, set tracker leader: 192.168.0.213:22122
[2012-03-13 08:38:05] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-03-13 08:38:34] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000[2012-03-25 18:11:06] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.212:22122, set tracker leader: 192.168.0.212:22122
[2012-03-25 18:11:08] ERROR - file: tracker_client_thread.c, line: 985, tracker server 192.168.0.213:22122, recv data fail, errno: 110, error info: Connection timed out.
[2012-03-25 18:11:14] ERROR - file: tracker_client_thread.c, line: 271, connect to tracker server 192.168.0.213:22122 fail, errno: 111, error info: Connection refused
[2012-03-25 18:11:44] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, continuous fail count: 1, as a tracker client, my ip is 192.168.0.245
[2012-03-25 18:12:14] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
[2012-05-04 01:54:09] INFO - FastDFS v3.05, base_path=/home/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=1024, work_threads=8, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, disk_rw_direct=0, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=
[2012-05-04 01:54:09] INFO - file: storage_param_getter.c, line: 97, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=102400 MB, use_trunk_file=0, slot_min_size=0, slot_max_size=16 MB, trunk_file_size=64 MB
[2012-05-04 01:54:09] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.213:22122, as a tracker client, my ip is 192.168.0.245
[2012-05-04 01:54:10] INFO - file: tracker_client_thread.c, line: 304, successfully connect to tracker server 192.168.0.212:22122, as a tracker client, my ip is 192.168.0.245
[2012-05-04 01:54:10] INFO - file: tracker_client_thread.c, line: 1100, tracker server 192.168.0.212:22122, set tracker leader: 192.168.0.212:22122[2012-05-04 01:54:39] INFO - file: storage_sync.c, line: 2485, successfully connect to storage server 192.168.0.11:23000
~