Hadoop的未来在哪里

wenhq

1、MapReduce的主要应用领域在哪里？在哪些场合被取代可能性不高？
a. offline computing. batch computing
b. query data with SQL?
c. it's very hard to be replace under batch processing.
2、对比YARN和Mesos的优势和劣势，以及YARN框架未来的发展方向？
a. Yarn support capacity/fair scheduler on memory/cpu which has fine-grained scheduler.
b. Mesos supprot coarse-grained scheduler which support yarn job also with non-yarn job.
3、HDFS缺少哪些你需要的特性，或者你比较喜欢其哪一个特性，也可以谈谈您比较看好哪个存储系统，为什么？
a.I like hdfs easily scaling. has default 3 replication with high availability. also it's take server down as common problems ,also build  on commodity server.reduce server-farm cost.
b.  compare to Glusterfs, Hdfs balance doesn't have high impact than GlusterFS.
c.  compare to Fastdfs, I thought it's can commit data replication more accurate than Fastdfs, which it is very hard under high volume write situation.
d. but, hdfs sync between cluster/DC. we have to use distcp tools  to make it, doesn't like NFS which need sync data easily.
e. hdfs doesn't like new tech ignite/tachyon which support memory-based storage will provide more faster access data, as it's data store on disk. you know, Disk I/O is  always bottleneck of performance.
4、Hadoop从业者应该如何进行职业规划？
Hadoop is a big ecosystem include storage/database/processing/security. I thought it's better do some project/experience under some mentor if possible. also you have to strong java coding skill, as it's based on java. after you did some projects, then try to understand the principle of Hadoop.
try to fix some bugs under github/googlegroup.  the most important part, you have to keep hungry till to understand the truth of Hadoop.

Just part of my opinion.

heguangwu

现在大的公司都逐步采用Spark了，当然并不是说不用MapReduce，当前情况是并存
回复 10# Steddywr


   

heguangwu

哪个新版本好像已经不再继续支持MapReduce了？
hadoop本身只有hdfs能持久，你认为yarn本身有什么问题吗？MapReduce只是Hadoop的一个内置分布式计算框架，Hadoop本身是可以支持多种计算架构，如spark、storm
回复 11# cokeboL


   

heguangwu

HDFS supported  Heterogeneous Storage Tiers since 2.6, for SSD storage tier, Memory as a storage tier (beta).
Hdfs balance is  node-based,not disk-based

回复 12# wenhq


   

cokeboL

回复 14# heguangwu


    忘记是hadoop原版还是cdh的某版本了，偶尔瞟到过一眼，不在继续支持是说不再继续改进或支持已有功能的bug之类，不是从功能里把MR去掉
    hadoop支持其他的我知道，就是因为支持spark storm，而且与他们相比，MR太鸡肋，所以没必要再用了

phobos

先占个位置

heguangwu

MapReduce如果架构不变，基本没有改进空间了，这也就是为什么hortonworks搞Tez的原因
回复 16# cokeboL


   

wenhq

回复 15# heguangwu

But it's rarely use Heterogeneous Storage？ right,
seems it's has Heterogeneous Storage feature during hadoop 2.6


btw, hadoop 2.7 support a new migirate tools like mover. which is used for archiving.

seems spark on yarn has lots of restriction ,especially in latest spark 1.6....need more time/effort to make it stable.


   

wenhq

回复 18# heguangwu


seems you are using tez?haha
Are u fans of Hortonworks?

   

wenhq

heguangwu 发表于 2016-01-25 14:07
哪个新版本好像已经不再继续支持MapReduce了？
hadoop本身只有hdfs能持久，你认为yarn本身有什么问题吗？M ...

I have some concern about yarn , on scheduler even u using fair scheduler/ capacity scheduler ,there still have some situation can't be meet.
eg: if I have A,B,C,D four Queue by using capacity scheduler,
and the  assginment is A 10% B 20% C 40% D 40% witch each setting maximum capacity is 90%, If queue C used absolute capacity 80%, and now another job coming to Queue D, looks like it won't running till there have enough resource available.