免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
楼主: Send_linux

[FastDFS] FastDFS一个高效的分布式文件系统 [复制链接]

论坛徽章:
0
发表于 2011-05-27 14:21 |显示全部楼层
你好,请问FastDFS的负载均衡是怎么实现的?是client请求读写(下载/上传)一个文件的时候,tracker server将其定位到负载最轻的storage server上吗?这样坐会不会造成那台服务器出现热点问题,这一块你们是怎么考虑的呢?谢谢~~

论坛徽章:
0
发表于 2012-08-28 14:24 |显示全部楼层
FastDFS能否支持跨地域的同步方式,例如同一个group下面的storage分布在不同地址位置的不同机房内,例如一个名叫default的group,组成此group的storage有两台,一台在广州,一台在北京,这样的方式支持吗?如果不支持,有没有内置的机制来完成多机房的容灾?或者,FastDFS是否提供日志机制,以便将一个机房内的文件修改同步到另外一个远程的机房?

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2012-08-28 16:42 |显示全部楼层
回复 32# anderson_gwr

FastDFS完全支持跨机房部署!
只要你机房之间网络是可以正常连接,就没问题。

论坛徽章:
0
发表于 2012-08-30 15:27 |显示全部楼层
因为机房之间的网络通常不稳定,经常出现网络波动,例如网络中断10分钟甚至1个小时,这个时候,FastDFS还能够正常工作吗?

另外,看FastDFS的Client API,发现文件名都是平面型的,即groupname/filename,这样的话,当保存大量文件的时候,在一个目录下面就会有着太多的文件存在。
有没有办法在storage path下面,自由地指定文件的目录,例如groupname/dir1/dir2/dir3/filename?

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2012-08-30 17:06 |显示全部楼层
回复 34# anderson_gwr

FastDFS存储文件采用的是256 * 256的两级目录啊!

论坛徽章:
0
发表于 2012-08-30 18:15 |显示全部楼层
嗯,对,FastDFS是两级目录,256 *256个子目录的。我的另外一个问题是:

因为机房之间的网络通常不稳定,经常出现网络波动,例如网络中断10分钟甚至1个小时,这个时候,FastDFS还能够正常工作吗?

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2012-08-31 10:06 |显示全部楼层
回复 36# anderson_gwr

没有问题的。FastDFS文件同步采用异步的增量方式。
网络恢复后,可以继续进行文件同步。

论坛徽章:
0
发表于 2012-08-31 14:47 |显示全部楼层
happy_fish, 非常感谢你的回答。我已经按照文档在三台虚拟机上搭建了fastdfs cluster。其中一台运行tracker, 另外两台运行storage。然后通过下面的命令上传:

xxx@ubuntu-1:/usr/local/bin$ sudo ./fdfs_upload_file /etc/fastdfs/client.conf ~/soft/memcached-1.4.14.tar.gz
realname/M00/00/00/rBMi01BAR92AzW1VAATm7i-gZic.tar.gz

结果发现只在一个storage server上面有上传的文件,另外一台storage上面不存在。查看trakcer的日志,有如下内容,不知道是什么原因导致的:

drutt@ubuntu-1:/usr/local/bin$ cat /opt/fastdfs/logs/trackerd.log
[2012-08-31 12:01:43] INFO - FastDFS v3.11, base_path=/opt/fastdfs, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=4096MB, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=10s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=3600s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0,
[2012-08-31 12:01:43] INFO - local_host_ip_count: 2,  127.0.0.1  172.19.34.210
[2012-08-31 12:05:49] CRIT - file: fdfs_trackerd.c, line: 413, catch signal 3, program exiting...
[2012-08-31 12:05:50] INFO - exit nomally.

[2012-08-31 12:06:48] INFO - FastDFS v3.11, base_path=/opt/fastdfs, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=4096MB, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=10s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=3600s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0,
[2012-08-31 12:06:48] INFO - HTTP supported: server_port=8080, default_content_type=application/octet-stream, anti_steal_token=0, token_ttl=0s, anti_steal_secret_key length=0, token_check_fail content_type=, token_check_fail buff length=0, check_active_interval=30, check_active_type=tcp, check_active_uri=/status.html
[2012-08-31 12:06:48] INFO - local_host_ip_count: 2,  127.0.0.1  172.19.34.210
[2012-08-31 12:52:10] INFO - file: tracker_relationship.c, line: 374, selecting leader...
[2012-08-31 12:52:10] INFO - file: tracker_relationship.c, line: 392, I am the new tracker leader 172.19.34.210:22122
[2012-08-31 12:52:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:52:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:53:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:53:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:54:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:54:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:55:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:55:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:56:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:56:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:57:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:57:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:58:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:58:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:59:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 12:59:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:00:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:00:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:01:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:01:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:02:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:02:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:03:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:03:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:04:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:04:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:05:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:05:52] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:06:22] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:08:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:09:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:09:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:09:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:09:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:10:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:10:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:10:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:10:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:11:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:11:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:11:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:11:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:12:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:12:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:12:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:12:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:13:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:13:13] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:13:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:13:43] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:14:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:14:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:14:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:14:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:15:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:15:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:15:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:15:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:16:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:16:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:16:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:16:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:17:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:17:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:17:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:17:44] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!
[2012-08-31 13:18:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.211:23000 idle too long, status change to offline!
[2012-08-31 13:18:14] INFO - file: tracker_mem.c, line: 5333, storage server 172.19.34.212:23000 idle too long, status change to offline!

论坛徽章:
4
2015年辞旧岁徽章
日期:2015-03-03 16:54:152015年迎新春徽章
日期:2015-03-04 09:56:11IT运维版块每日发帖之星
日期:2016-08-11 06:20:00IT运维版块每日发帖之星
日期:2016-08-15 06:20:00
发表于 2012-08-31 17:42 |显示全部楼层
回复 38# anderson_gwr

你修改过tracker.conf中的一些参数吧。

tracker.conf:
# check storage server alive interval seconds
check_active_interval = 120

storage.conf:
# heart beat interval in seconds
heart_beat_interval=30


对于这两个参数的大小关系,必须是:check_active_interval  > heart_beat_interval
如果你不修改这两个参数值,是没有任何问题的啊!
请检查一下这两个配置项。

论坛徽章:
0
发表于 2012-11-02 17:29 |显示全部楼层
回复 39# happy_fish100



请问:track服务器内存需要多大?


   
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

SACC2019中国系统架构师大会

【数字转型 架构演进】SACC2019中国系统架构师大会,7折限时优惠重磅来袭!
2019年10月31日~11月2日第11届中国系统架构师大会(SACC2019)将在北京隆重召开。四大主线并行的演讲模式,1个主会场、20个技术专场、超千人参与的会议规模,100+来自互联网、金融、制造业、电商等领域的嘉宾阵容,将为广大参会者提供一场最具价值的技术交流盛会。

限时七折期:2019年8月31日前


----------------------------------------

大会官网>>
  

北京盛拓优讯信息技术有限公司. 版权所有 16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122
中国互联网协会会员  联系我们:huangweiwei@it168.com
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP