论坛徽章:: 0

电梯直达

1楼 [收藏(0)] [报告]

发表于 2011-12-22 08:52 |只看该作者 |倒序浏览

这里省略hadoop的介绍，直接介绍安装步骤，按照这步骤就能克隆搭建一个实例。 角色列表： namenode & jobtracker 192.168.237.13 datanode & tasktracker 192.168.237.74 datanode & tasktracker 192.168.239.128 #useradd hadoop download hadoop-0.20.2.tar.gz  http://mirror.bjtu.edu.cn/apache/hadoop/core/hadoop-0.20.2/ #mkdir /data/hadoop #tar -zxvf hadoop-0.20.2.tar.gz #chown -R hadoop:hadoop hadoop-0.20.2 hadoop 解决无密码登录问题 #./ssh_nopasswd.sh client && ./ssh_nopasswd.sh server 按需修改用户和路径 <a href=".http://blog.chinaunix.net/attachment/attach/22/27/07/73222707737ff3e4021253d530696d46f60467d238.zip" target="_blank" target="_blank"><img src="/blog/image/attachicons/zip.gif" align="absmiddle" border="0"> ssh_nopasswd.zip </a>  ---------------------------- 以下四个文件的配置，在一台机上编辑好后，传到其它机器上，面前重复编辑。 相关文件配置: core-site.xml 配置namenode jobtracker基本信息 主要配置 fs.default.name：URI of NameNode mapred.job.tracker：jobtracker ip 和端口 hadoop.tmp.dir：hadoop临时目录 dfs.name.dir：name table存储路径 dfs.data.dir：namenode数据块配置 dfs.replication：副本数 PS： 我的host中进行了如下设置： 192.168.237.13  hadoop-237-13.pconline.ctc      hadoop-237-13 192.168.237.74  hadoop-237-74.pconline.ctc      hadoop-237-74 192.168.239.128  hadoop-239-128.pconline.ctc      hadoop-239-128 例子： <property> <name>fs.default.name</name> <value>hdfs://hadoop-237-13:9000</value> <description>The name of the default file system. Either the literal string "local" or a host:port for DFS.</description> </property> <property> <name>mapred.job.tracker</name> <value>192.168.237.13:9001</value> <description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.</description> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/hadoop/tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>dfs.name.dir</name> <value>/data/hadoop/filesystem/name</value> <description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description> </property> <property> <name>dfs.data.dir</name> <value>/data/hadoop/filesystem/data</value> <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are i gnored.</description> </property> <property> <name>dfs.replication</name> <value>2</value> <description>Default block replication. The actual number of replications can be specified when the file is created. The default isused if replication is not specified in create time.</description> mapred-site.xml 配置map reduce 的一些细节信息 看description进行配置就行 <property> <name>mapred.job.tracker</name> <value>192.168.237.13:9001</value> <description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.</description> </property> <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>2</value> <description>The maximum number of map tasks that will be run simultaneously by a task tracker.</description> </property> <property> <name>mapred.tasktracker.reduce.tasks.maximum</name> <value>2</value> <description>The maximum number of reduce tasks that will be run simultaneously by a task tracker.</description> </property> <property> <name>mapred.map.tasks</name> <value>2</value> <description>The default number of map tasks per job. Ignored when mapred.job.tracker is "local".</description> </property> <property> <name>mapred.reduce.tasks</name> <value>2</value> <description>The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapred.job.tracker is "local".</description> </property> <property> <name>mapred.userlog.retain.hours</name> <value>2</value> <description>The maximum time, in hours, for which the user-logs are to be retained.</description> </property> <property>    <name>mapred.child.java.opts</name>    <value>-Xmx700M -server</value> </property> <property>   <name>mapred.map.max.attempts</name>   <value>800</value>   <description>Expert: The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it.</description> </property> <property>   <name>mapred.reduce.max.attempts</name>   <value>800</value>   <description>Expert: The maximum number of attempts per reduce task. In other words, framework will try to execute a reduce task these many number of times before giving up on it.</description> </property> <property>   <name>mapred.max.tracker.failures</name>   <value>800</value>   <description>The number of task-failures on a tasktracker of a given job after which new tasks of that job aren't assigned to it. </description> </property> <property>   <name>mapred.task.timeout</name>   <value>60000000</value>   <description>The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. </description> </property> masters secondarynamenode：这里只为测试只做在namenode本机上了 里面信息为 192.168.237.13 slaves: 里面信息为： 192.168.237.74 192.168.239.128 因为我机器中没配置JAVA_HOME环境变量，所以在hadoop-env.sh文件中进行设置 export JAVA_HOME=/usr/java/jdk1.6.0_22 ---------------------------- #cd /datat/hadoop && su hadoop $bin/hadoop namenode -format $bin/start-all.sh $ bin/hadoop dfsadmin -report 显示如下信息，为成功。 Configured Capacity: 107981234176 (100.57 GB) Present Capacity: 101694681088 (94.71 GB) DFS Remaining: 101694607360 (94.71 GB) DFS Used: 73728 (72 KB) DFS Used%: 0% Under replicated blocks: 1 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Datanodes available: 2 (2 total, 0 dead) Name: 192.168.239.128:50010 Decommission Status : Normal Configured Capacity: 53558603776 (49.88 GB) DFS Used: 36864 (36 KB) Non DFS Used: 3143274496 (2.93 GB) DFS Remaining: 50415292416(46.95 GB) DFS Used%: 0% DFS Remaining%: 94.13% Last contact: Fri Aug 05 12:19:33 CST 2011 Name: 192.168.237.74:50010 Decommission Status : Normal Configured Capacity: 54422630400 (50.69 GB) DFS Used: 36864 (36 KB) Non DFS Used: 3143278592 (2.93 GB) DFS Remaining: 51279314944(47.76 GB) DFS Used%: 0% DFS Remaining%: 94.22% Last contact: Fri Aug 05 12:19:33 CST 2011 安装过程中的一些错误： 1./data/hadoop 没有做chown  提示没权限 2./data/hadoop中手工创建了tmp data相关目录，提示 2011-08-05 09:40:34,559 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://hadoop-237-13:9000/data/hadoop/tmp/mapred/system org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /data/hadoop/tmp/mapred/system. Name node is in safe mode. 如果遇到错误，多查看hadoop_home/logs下来相关信息 参考信息： http://hadoop.apache.org/common/docs/r0.21.0/cluster_setup.html

文库|博客

返回列表

Chinaunix › 论坛 › 数据库技术 › NoSQL技术 › hadoop 快速安装

[Hadoop&HBase] hadoop 快速安装 [复制链接]