Hadoop的未来在哪里

cokeboL

1、MapReduce的主要应用领域在哪里？在哪些场合被取代可能性不高？
     主要应用在大数据处理、分析。新版本好像已经不再继续支持MapReduce了，性能相比Spark之类太差，被Spark取代几乎是必然了。
2、对比YARN和Mesos的优势和劣势，以及YARN框架未来的发展方向？
     Yarn和Mesos我都没怎么用，在hadoop几个技术群里聊，Mesos似乎更优秀，也是未来大势所趋的样子。
3、HDFS缺少哪些你需要的特性，或者你比较喜欢其哪一个特性，也可以谈谈您比较看好哪个存储系统，为什么？
     缺少的特性就是，我不喜欢java，希望能对python提供编程接口，虽然streaming可以使用各种语言处理大数据，但是文件系统相关的
     编程接口只支持java，也是很虐心。功能上，作为分布式文件系统，还好了，挺全面的没遇到什么缺的。喜欢的特性，当然是命令和shell
     很像，容易入门和使用
4、Hadoop从业者应该如何进行职业规划？
     个人觉得hadoop本身只有hdfs能持久，MR会逐渐被Spark取代，hdfs相关的业务，可能只是两大部分：一是运维， 二是存储相关的开发。
     这两个方向的iter自然需要去学习和实战相关知识。
     另外，各大行业数据相关的业务，如数据处理、分析的人员需求会更多些，而需要的专业知识和算法要求也更高些，可以在数据挖掘上多深入
     研究下。

wenhq

1、MapReduce的主要应用领域在哪里？在哪些场合被取代可能性不高？
a. offline computing. batch computing
b. query data with SQL?
c. it's very hard to be replace under batch processing.
2、对比YARN和Mesos的优势和劣势，以及YARN框架未来的发展方向？
a. Yarn support capacity/fair scheduler on memory/cpu which has fine-grained scheduler.
b. Mesos supprot coarse-grained scheduler which support yarn job also with non-yarn job.
3、HDFS缺少哪些你需要的特性，或者你比较喜欢其哪一个特性，也可以谈谈您比较看好哪个存储系统，为什么？
a.I like hdfs easily scaling. has default 3 replication with high availability. also it's take server down as common problems ,also build  on commodity server.reduce server-farm cost.
b.  compare to Glusterfs, Hdfs balance doesn't have high impact than GlusterFS.
c.  compare to Fastdfs, I thought it's can commit data replication more accurate than Fastdfs, which it is very hard under high volume write situation.
d. but, hdfs sync between cluster/DC. we have to use distcp tools  to make it, doesn't like NFS which need sync data easily.
e. hdfs doesn't like new tech ignite/tachyon which support memory-based storage will provide more faster access data, as it's data store on disk. you know, Disk I/O is  always bottleneck of performance.
4、Hadoop从业者应该如何进行职业规划？
Hadoop is a big ecosystem include storage/database/processing/security. I thought it's better do some project/experience under some mentor if possible. also you have to strong java coding skill, as it's based on java. after you did some projects, then try to understand the principle of Hadoop.
try to fix some bugs under github/googlegroup.  the most important part, you have to keep hungry till to understand the truth of Hadoop.

Just part of my opinion.

heguangwu

现在大的公司都逐步采用Spark了，当然并不是说不用MapReduce，当前情况是并存
回复 10# Steddywr


   

heguangwu

哪个新版本好像已经不再继续支持MapReduce了？
hadoop本身只有hdfs能持久，你认为yarn本身有什么问题吗？MapReduce只是Hadoop的一个内置分布式计算框架，Hadoop本身是可以支持多种计算架构，如spark、storm
回复 11# cokeboL


   

heguangwu

HDFS supported  Heterogeneous Storage Tiers since 2.6, for SSD storage tier, Memory as a storage tier (beta).
Hdfs balance is  node-based,not disk-based

回复 12# wenhq


   

cokeboL

回复 14# heguangwu


    忘记是hadoop原版还是cdh的某版本了，偶尔瞟到过一眼，不在继续支持是说不再继续改进或支持已有功能的bug之类，不是从功能里把MR去掉
    hadoop支持其他的我知道，就是因为支持spark storm，而且与他们相比，MR太鸡肋，所以没必要再用了

phobos

先占个位置

heguangwu

MapReduce如果架构不变，基本没有改进空间了，这也就是为什么hortonworks搞Tez的原因
回复 16# cokeboL


   

wenhq

回复 15# heguangwu

But it's rarely use Heterogeneous Storage？ right,
seems it's has Heterogeneous Storage feature during hadoop 2.6


btw, hadoop 2.7 support a new migirate tools like mover. which is used for archiving.

seems spark on yarn has lots of restriction ,especially in latest spark 1.6....need more time/effort to make it stable.


   

wenhq

回复 18# heguangwu


seems you are using tez?haha
Are u fans of Hortonworks?