免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
12下一页
最近访问板块 发新帖
查看: 11220 | 回复: 13

[FastDFS] FastDFS的storager server如何手动同步? [复制链接]

论坛徽章:
1
摩羯座
日期:2015-01-08 14:01:55
发表于 2014-05-15 18:09 |显示全部楼层
请教一下,storager server如何手动同步呢?谢谢!
我用的FastDFS_v4.06,一共两个group,奇怪的是storager server无法同步之前的数据。
期间更换过一台tracker server和一台storager server。
在更换tracker server之前,我测试过,可以同步历史数据到新增storager的服务器上面,但是现在无法同步上历史数据,只能同步新增的数据。

以下是fdfs_monitor所获取的信息,last_source_update、last_sync_update和last_synced_timestamp的值都为1970-01-01 08:00:00。
  1. [2014-05-15 10:01:34] DEBUG - base_path=/usr/local/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

  2. server_count=1, server_index=0

  3. tracker server is 172.16.0.60:22122

  4. group count: 2

  5. Group 1:
  6. group name = group1
  7. disk total space = 4031 MB
  8. disk free space = 2452 MB
  9. trunk free space = 0 MB
  10. storage server count = 2
  11. active server count = 2
  12. storage server port = 23000
  13. storage HTTP port = 8090
  14. store path count = 1
  15. subdir count per path = 256
  16. current write server index = 0
  17. current trunk file id = 0

  18.         Storage 1:
  19.                 id = 172.16.0.56
  20.                 ip_addr = 172.16.0.56  ACTIVE
  21.                 http domain =
  22.                 version = 4.06
  23.                 join time = 2014-05-15 09:48:41
  24.                 up time = 2014-05-15 09:48:41
  25.                 total storage = 4031 MB
  26.                 free storage = 2452 MB
  27.                 upload priority = 10
  28.                 store_path_count = 1
  29.                 subdir_count_per_path = 256
  30.                 storage_port = 23000
  31.                 storage_http_port = 8090
  32.                 current_write_path = 0
  33.                 source storage id= 172.16.0.59
  34.                 if_trunk_server= 0
  35.                 total_upload_count = 0
  36.                 success_upload_count = 0
  37.                 total_append_count = 0
  38.                 success_append_count = 0
  39.                 total_modify_count = 0
  40.                 success_modify_count = 0
  41.                 total_truncate_count = 0
  42.                 success_truncate_count = 0
  43.                 total_set_meta_count = 0
  44.                 success_set_meta_count = 0
  45.                 total_delete_count = 0
  46.                 success_delete_count = 0
  47.                 total_download_count = 0
  48.                 success_download_count = 0
  49.                 total_get_meta_count = 0
  50.                 success_get_meta_count = 0
  51.                 total_create_link_count = 0
  52.                 success_create_link_count = 0
  53.                 total_delete_link_count = 0
  54.                 success_delete_link_count = 0
  55.                 total_upload_bytes = 0
  56.                 success_upload_bytes = 0
  57.                 total_append_bytes = 0
  58.                 success_append_bytes = 0
  59.                 total_modify_bytes = 0
  60.                 success_modify_bytes = 0
  61.                 stotal_download_bytes = 0
  62.                 success_download_bytes = 0
  63.                 total_sync_in_bytes = 0
  64.                 success_sync_in_bytes = 0
  65.                 total_sync_out_bytes = 0
  66.                 success_sync_out_bytes = 0
  67.                 total_file_open_count = 0
  68.                 success_file_open_count = 0
  69.                 total_file_read_count = 0
  70.                 success_file_read_count = 0
  71.                 total_file_write_count = 0
  72.                 success_file_write_count = 0
  73.                 last_heart_beat_time = 2014-05-15 10:01:10
  74.                 last_source_update = 1970-01-01 08:00:00
  75.                 last_sync_update = 1970-01-01 08:00:00
  76.                 last_synced_timestamp = 1970-01-01 08:00:00
  77.         Storage 2:
  78.                 id = 172.16.0.59
  79.                 ip_addr = 172.16.0.59  ACTIVE
  80.                 http domain =
  81.                 version = 4.06
  82.                 join time = 2014-05-15 09:48:28
  83.                 up time = 2014-05-15 09:48:28
  84.                 total storage = 4031 MB
  85.                 free storage = 2489 MB
  86.                 upload priority = 10
  87.                 store_path_count = 1
  88.                 subdir_count_per_path = 256
  89.                 storage_port = 23000
  90.                 storage_http_port = 8090
  91.                 current_write_path = 0
  92.                 source storage id=
  93.                 if_trunk_server= 0
  94.                 total_upload_count = 0
  95.                 success_upload_count = 0
  96.                 total_append_count = 0
  97.                 success_append_count = 0
  98.                 total_modify_count = 0
  99.                 success_modify_count = 0
  100.                 total_truncate_count = 0
  101.                 success_truncate_count = 0
  102.                 total_set_meta_count = 0
  103.                 success_set_meta_count = 0
  104.                 total_delete_count = 0
  105.                 success_delete_count = 0
  106.                 total_download_count = 0
  107.                 success_download_count = 0
  108.                 total_get_meta_count = 0
  109.                 success_get_meta_count = 0
  110.                 total_create_link_count = 0
  111.                 success_create_link_count = 0
  112.                 total_delete_link_count = 0
  113.                 success_delete_link_count = 0
  114.                 total_upload_bytes = 0
  115.                 success_upload_bytes = 0
  116.                 total_append_bytes = 0
  117.                 success_append_bytes = 0
  118.                 total_modify_bytes = 0
  119.                 success_modify_bytes = 0
  120.                 stotal_download_bytes = 0
  121.                 success_download_bytes = 0
  122.                 total_sync_in_bytes = 0
  123.                 success_sync_in_bytes = 0
  124.                 total_sync_out_bytes = 0
  125.                 success_sync_out_bytes = 0
  126.                 total_file_open_count = 0
  127.                 success_file_open_count = 0
  128.                 total_file_read_count = 0
  129.                 success_file_read_count = 0
  130.                 total_file_write_count = 0
  131.                 success_file_write_count = 0
  132.                 last_heart_beat_time = 2014-05-15 10:01:27
  133.                 last_source_update = 1970-01-01 08:00:00
  134.                 last_sync_update = 1970-01-01 08:00:00
  135.                 last_synced_timestamp = 1970-01-01 08:00:00

  136. Group 2:
  137. group name = group2
  138. disk total space = 4031 MB
  139. disk free space = 2487 MB
  140. trunk free space = 0 MB
  141. storage server count = 2
  142. active server count = 2
  143. storage server port = 23000
  144. storage HTTP port = 8090
  145. store path count = 1
  146. subdir count per path = 256
  147. current write server index = 0
  148. current trunk file id = 0

  149.         Storage 1:
  150.                 id = 172.16.0.54
  151.                 ip_addr = 172.16.0.54  ACTIVE
  152.                 http domain =
  153.                 version = 4.06
  154.                 join time = 2014-05-15 09:49:22
  155.                 up time = 2014-05-15 09:49:22
  156.                 total storage = 4031 MB
  157.                 free storage = 2487 MB
  158.                 upload priority = 10
  159.                 store_path_count = 1
  160.                 subdir_count_per_path = 256
  161.                 storage_port = 23000
  162.                 storage_http_port = 8090
  163.                 current_write_path = 0
  164.                 source storage id= 172.16.0.55
  165.                 if_trunk_server= 0
  166.                 total_upload_count = 0
  167.                 success_upload_count = 0
  168.                 total_append_count = 0
  169.                 success_append_count = 0
  170.                 total_modify_count = 0
  171.                 success_modify_count = 0
  172.                 total_truncate_count = 0
  173.                 success_truncate_count = 0
  174.                 total_set_meta_count = 0
  175.                 success_set_meta_count = 0
  176.                 total_delete_count = 0
  177.                 success_delete_count = 0
  178.                 total_download_count = 0
  179.                 success_download_count = 0
  180.                 total_get_meta_count = 0
  181.                 success_get_meta_count = 0
  182.                 total_create_link_count = 0
  183.                 success_create_link_count = 0
  184.                 total_delete_link_count = 0
  185.                 success_delete_link_count = 0
  186.                 total_upload_bytes = 0
  187.                 success_upload_bytes = 0
  188.                 total_append_bytes = 0
  189.                 success_append_bytes = 0
  190.                 total_modify_bytes = 0
  191.                 success_modify_bytes = 0
  192.                 stotal_download_bytes = 0
  193.                 success_download_bytes = 0
  194.                 total_sync_in_bytes = 0
  195.                 success_sync_in_bytes = 0
  196.                 total_sync_out_bytes = 0
  197.                 success_sync_out_bytes = 0
  198.                 total_file_open_count = 0
  199.                 success_file_open_count = 0
  200.                 total_file_read_count = 0
  201.                 success_file_read_count = 0
  202.                 total_file_write_count = 0
  203.                 success_file_write_count = 0
  204.                 last_heart_beat_time = 2014-05-15 10:01:20
  205.                 last_source_update = 1970-01-01 08:00:00
  206.                 last_sync_update = 1970-01-01 08:00:00
  207.                 last_synced_timestamp = 1970-01-01 08:00:00
  208.         Storage 2:
  209.                 id = 172.16.0.55
  210.                 ip_addr = 172.16.0.55  ACTIVE
  211.                 http domain =
  212.                 version = 4.06
  213.                 join time = 2014-05-15 09:49:04
  214.                 up time = 2014-05-15 09:49:04
  215.                 total storage = 4031 MB
  216.                 free storage = 2510 MB
  217.                 upload priority = 10
  218.                 store_path_count = 1
  219.                 subdir_count_per_path = 256
  220.                 storage_port = 23000
  221.                 storage_http_port = 8090
  222.                 current_write_path = 0
  223.                 source storage id=
  224.                 if_trunk_server= 0
  225.                 total_upload_count = 0
  226.                 success_upload_count = 0
  227.                 total_append_count = 0
  228.                 success_append_count = 0
  229.                 total_modify_count = 0
  230.                 success_modify_count = 0
  231.                 total_truncate_count = 0
  232.                 success_truncate_count = 0
  233.                 total_set_meta_count = 0
  234.                 success_set_meta_count = 0
  235.                 total_delete_count = 0
  236.                 success_delete_count = 0
  237.                 total_download_count = 0
  238.                 success_download_count = 0
  239.                 total_get_meta_count = 0
  240.                 success_get_meta_count = 0
  241.                 total_create_link_count = 0
  242.                 success_create_link_count = 0
  243.                 total_delete_link_count = 0
  244.                 success_delete_link_count = 0
  245.                 total_upload_bytes = 0
  246.                 success_upload_bytes = 0
  247.                 total_append_bytes = 0
  248.                 success_append_bytes = 0
  249.                 total_modify_bytes = 0
  250.                 success_modify_bytes = 0
  251.                 stotal_download_bytes = 0
  252.                 success_download_bytes = 0
  253.                 total_sync_in_bytes = 0
  254.                 success_sync_in_bytes = 0
  255.                 total_sync_out_bytes = 0
  256.                 success_sync_out_bytes = 0
  257.                 total_file_open_count = 0
  258.                 success_file_open_count = 0
  259.                 total_file_read_count = 0
  260.                 success_file_read_count = 0
  261.                 total_file_write_count = 0
  262.                 success_file_write_count = 0
  263.                 last_heart_beat_time = 2014-05-15 10:01:32
  264.                 last_source_update = 1970-01-01 08:00:00
  265.                 last_sync_update = 1970-01-01 08:00:00
  266.                 last_synced_timestamp = 1970-01-01 08:00:00
复制代码
group1中的一台storager server日志如下。
  1. [2014-05-15 09:48:28] INFO - FastDFS v4.06, base_path=/usr/local/fastdfs, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8090, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
  2. [2014-05-15 09:48:29] INFO - file: storage_param_getter.c, line: 187, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, store_slave_file_use_link=0
  3. [2014-05-15 09:48:29] INFO - file: storage_func.c, line: 174, tracker_client_ip: 172.16.0.59, my_server_id_str: 172.16.0.59, g_server_id_in_filename: 989860012
  4. [2014-05-15 09:48:29] INFO - local_host_ip_count: 2,  127.0.0.1  172.16.0.59
  5. [2014-05-15 09:48:29] INFO - file: tracker_client_thread.c, line: 308, successfully connect to tracker server 172.16.0.60:22122, as a tracker client, my ip is 172.16.0.59
  6. [2014-05-15 09:48:59] INFO - file: tracker_client_thread.c, line: 1124, tracker server 172.16.0.60:22122, set tracker leader: 172.16.0.60:22122
  7. [2014-05-15 09:48:59] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 172.16.0.56:23000
  8. [2014-05-15 09:49:29] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 172.16.0.56:23000
复制代码
tracker server的日志也一并贴上来。
  1. [2014-05-15 09:48:06] INFO - FastDFS v4.06, base_path=/usr/local/fastdfs, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, work_threads=4, store_lookup=2, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
  2. [2014-05-15 09:48:06] INFO - local_host_ip_count: 2,  127.0.0.1  172.16.0.60
  3. [2014-05-15 09:48:27] INFO - file: tracker_relationship.c, line: 374, selecting leader...
  4. [2014-05-15 09:48:28] INFO - file: tracker_relationship.c, line: 392, I am the new tracker leader 172.16.0.60:22122
复制代码

论坛徽章:
0
发表于 2014-05-19 16:36 |显示全部楼层
我也碰到这个问题, 请问happy fish, 这个怎么解决啊?

论坛徽章:
0
发表于 2014-05-20 10:54 |显示全部楼层
解决方法是: 同组里的storage都重启一下, 如果还不行, tracker也都重启一下

论坛徽章:
1
摩羯座
日期:2015-01-08 14:01:55
发表于 2014-05-21 11:52 |显示全部楼层
回复 3# fxysun

感谢您的回复,您说的方法,我之前都试过了,依旧无法同步历史数据。
   

论坛徽章:
1
摩羯座
日期:2015-01-08 14:01:55
发表于 2014-05-21 11:54 |显示全部楼层
@happy_fish100
麻烦您有空的话,告知一下接近方法,谢谢。

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2014-05-22 16:01 |显示全部楼层
确认文件没有同步?

论坛徽章:
1
摩羯座
日期:2015-01-08 14:01:55
发表于 2014-05-23 11:10 |显示全部楼层
happy_fish100 发表于 2014-05-22 16:01
确认文件没有同步?


是的呀。
历史文件没有自动同步,而且很奇怪的是除了last_heart_beat_time是系统时间,其他三个选项都是1970.01.01.的时间戳。
请问能否实现手头同步历史文件呢?谢谢!
  1.                 last_source_update = 1970-01-01 08:00:00
  2.                 last_sync_update = 1970-01-01 08:00:00
  3.                 last_synced_timestamp = 1970-01-01 08:00:00
复制代码

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2014-05-23 17:39 |显示全部楼层
你后来上传过文件么?

论坛徽章:
1
摩羯座
日期:2015-01-08 14:01:55
发表于 2014-05-23 17:54 |显示全部楼层
happy_fish100 发表于 2014-05-23 17:39
你后来上传过文件么?


后来上传过文件,也可以同步后来上传的文件,但是历史文件无法同步。
场景的话,在帖子开头有说明:期间更换过一台tracker server和一台storager server。

论坛徽章:
0
发表于 2014-05-29 18:30 |显示全部楼层
重点在你是怎么换的,还有是否影响了binlog的完整性。回复 9# 可恶的


   
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

DTCC2020中国数据库技术大会

【架构革新 高效可控】2020年12月21日-23日第十一届中国数据库技术大会将在北京隆重召开。

大会设置2大主会场,20+技术专场,将邀请超百位行业专家,重点围绕数据架构、AI与大数据、传统企业数据库实践和国产开源数据库等内容展开分享和探讨,为广大数据领域从业人士提供一场年度盛会和交流平台。

http://dtcc.it168.com


大会官网>>
  

北京盛拓优讯信息技术有限公司. 版权所有 16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122
中国互联网协会会员  联系我们:huangweiwei@it168.com
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP