免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
楼主: shinelian

[mooseFS] ▇▇▇mfs权威指南(moosefs)分布式文件系统一站式解决方案(部署,性能测试)不断更新 [复制链接]

论坛徽章:
0
发表于 2010-01-20 20:43 |显示全部楼层
学习

论坛徽章:
0
发表于 2010-01-22 15:15 |显示全部楼层
先顶起啊, 这个真不错,已经开始搭建实验环境了 :wink:

论坛徽章:
0
发表于 2010-01-23 17:56 |显示全部楼层
很好的东东 不仅仅体现在MFS上

论坛徽章:
0
发表于 2010-01-26 10:23 |显示全部楼层

回复 #1 shinelian 的帖子

十分详细,学习ing...

论坛徽章:
0
发表于 2010-01-26 17:42 |显示全部楼层
非常不错mfs1.6了,,很不错的

论坛徽章:
8
综合交流区版块每周发帖之星
日期:2015-12-02 15:03:53数据库技术版块每日发帖之星
日期:2015-10-02 06:20:00IT运维版块每日发帖之星
日期:2015-10-02 06:20:00IT运维版块每日发帖之星
日期:2015-09-14 06:20:00金牛座
日期:2014-10-10 11:23:34CU十二周年纪念徽章
日期:2013-10-24 15:41:34酉鸡
日期:2013-10-19 10:17:1315-16赛季CBA联赛之北京
日期:2017-03-06 15:12:44
发表于 2010-01-26 18:23 |显示全部楼层
感觉非常不错

论坛徽章:
0
发表于 2010-02-02 21:40 |显示全部楼层
为什么要研究MFS?
在这有几个疑问:
1)楼主的测试环境有待提高。作为分布式存储环境,几十台,上百台,上千台很正常,楼主测试的网络环境、存储环境都有待提高,因此测试出来的数据借鉴性非常小。
2)MFS的优点在哪里?没有看出MFS到底好在什么地方,他有的,别人都有,别人有的,他有没有未必。再说,如果不支持几百个节点,这个文件系统就是失败的。
3)不要太迷信Google 文件系统。其实这个文件系统设计架构很一般,而且这个文件系统主要针对google的海量信息检索处理专门设计,因此,尤其独到之处,但在其他地方未必适用。
4)开源的分布式文件系统非常多,个人比较推荐Gluster,官方测试最高带宽达到32GB/S,号称比lustre还要好,和MFS相似之处是基于用户态的文件系统,也支持分条、副本,快照好像不支持,有很多性能优化的策略,配置也比较简单。

论坛徽章:
1
CU十二周年纪念徽章
日期:2013-10-24 15:41:34
发表于 2010-02-02 22:35 |显示全部楼层
不错

论坛徽章:
0
发表于 2010-02-03 17:47 |显示全部楼层
回复 27# sannas



    @sannas  首先看到如此有建设性的回帖 十分感动:

1)事实确实如此,本文在现阶段属于抛砖引玉之文章,不过我们的 分布式文件系统 专业qq群里面 ,有其他哥们做了更加有说服性的测试,包括性能测试,灾难恢复测试等等,希望你能加入我们的社区讨论 。

2)由于mfs是我测试的第一个分布式文件系统,可能有先入为主的概念,同样也是希望大家踊跃提出建设性的想法。目前我对比测试其他1,2个分布式文件系统,发现mfs在某些特性上是要强于它们的。

3)由于你的这个建议,我会花时间在好好分析一下google filesystem原理,以及适用范围。

4)前2天刚好在看这方面的资料,目前网络上有 分布式文件系统 专业qq群 里面 蚊子兄弟的一系列文章 ( http://www.wenzizone.cn/ ),看了感觉还不错,感觉它的分布式的内涵和mfs应该有所区别,由于你的建议,我同样会花时间去分析和测试下。

再次谢谢你的建议,希望再次和你交流。

论坛徽章:
0
发表于 2010-02-05 12:24 |显示全部楼层

更新ing 14. mfs master 热备方案:

本帖最后由 shinelian 于 2010-02-05 13:11 编辑

更新ing
14. mfs master 热备方案:

前2天问了mfs作者关于 性能测试和灾难恢复这2块的情况,如下是回复,随信还附了mini how to :
Hi!

We are very happy that you are a user of MooseFS! Also thank you very much
for promoting the system in China!

Regarding the Distaster Recovery please read the mini how which I attach -
you can easily change the metalogger machine to be the master server.

Regarding performance tests - we have not done any extensive tests. And we
are happy that people from open source community like you contribute to it
and fill the gap.

We are preparing our new website and we would like to put there
descriptions of installed MooseFS systems all over the world. Could you
write more about your architecture? What is your master server? How many
chunkservers do you have? What is the total space used by you? And so on
and so on - the more details, the better.


Kind regards
Michał Borychowski
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00;
Fax : +4822 874-41-01

#######################################################################3

How to prepare a fail proof solution with a redundant master?
A redundant master functionality is right now not a built-in functionality. But this subject is for us very crucial because we know how important this is and we receive lots of requests about it from many sources.

master的冗余能力目前还不是一个原生或者内建的功能,但这对我们来说是一个非常关键的课题,因为它是非常重要以及近来从许多地方受到了大量关于此问题的询问。

It is important to mention that even in MooseFS v 1.5.x edition it is relatively easy to write a scripts set which would quite automatically start a backup master server and in 1.6.x it is even simpler. The whole process of switching to the backup server would take less than a minute.

特别重要的,在moosefe1.5.x 系列已经可以通过编写脚本来自动的启动一个备援的master,在1.6.x系列则更为简单。切换到备援master的整个过程会花掉不到一分钟的时间。

It is enough to use for example Common Address Redundancy Protocol (http://www.openbsd.org/faq/pf/carp.html, http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol). CARP allows that there exist two machines with the same IP in one LAN - one MASTER and the second BACKUP.

So you can set up IP of mfsmaster on a CARP interface and configure the master machine to be used as MooseFS main master. On the backup machine you also install mfsmaster but of course you do not run it.

Versions 1.6.5 and above contain a new program "mfsmetalogger" which you can run on whatever machine you wish. The program gets metadata from master - every several hours (by default every 24 hours) gets a full metadata file and on current basis a complete change log.

If you run an earlier version of MooseFS than 1.6.5 it is enough to set up several simple scripts run regularly from cron (eg. every one hour) which would backup metadata file from the main master "PREFIX/var/mfs/metadata.mfs.back".

You also need an extra script run continuously testing the CARP interface state which in case if this interface goes in a MASTER mode would get two or three newest "changelog" files from any chunkserver (just by using "scp"), would also start "mfsmetarestore" and then "mfsmaster". The switch time should take approximately several seconds and along with time necessary for reconnecting chunkservers a new master would be fully functional in about a minute (in both read and write modes).

We also plan to add option to run master in read-only mode - which would be the best option for running the backup machine. This would secure the system against potential desynchronization of the two master machines and the need of merging all the changes to the main master which took place on the backup master.
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP