论坛徽章:: 27

电梯直达

1楼 [收藏(0)] [报告]

发表于 2011-12-21 08:42 |只看该作者 |倒序浏览

<div>一、前期准备:</div><div>下载hadoop: http://hadoop.apache.org/core/releases.html</div><div>http://hadoop.apache.org/common/releases.html</div><div>http://www.apache.org/dyn/closer.cgi/hadoop/core/</div><div>http://labs.xiaonei.com/apache-mirror/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz</div><div>http://labs.xiaonei.com/apache-mirror/hadoop/</div><div>二、硬件环境</div><div>共有3台机器，均使用的CentOS，Java使用的是jdk1.6.0。</div><div><br></div><div>三、安装JAVA6</div><div>sudo apt-get install sun-java6-jdk</div><div><br></div><div>/etc/environment</div><div>打开之后加入：#中间是以英文的冒号隔开，记得windows中是以英文的分号做为分隔的</div><div>CLASSPATH=.:/usr/local/java/lib</div><div>JAVA_HOME=/usr/local/java</div><div><br></div><div><br></div><div>三、配置host表</div><div>[root@hadoop ~]# vi /etc/hosts</div><div>127.0.0.1       localhost</div><div>192.168.13.100     namenode</div><div>192.168.13.108     datanode1</div><div>192.168.13.110     datanode2</div><div><br></div><div>[root@test ~]# vi /etc/hosts</div><div>127.0.0.1       localhost</div><div>192.168.13.100     namenode</div><div>192.168.13.108     datanode1</div><div><br></div><div>[root@test2 ~]# vi /etc/host</div><div>127.0.0.1       localhost</div><div>192.168.13.100     namenode</div><div>192.168.13.110     datanode2</div><div>添加用户和用户组</div><div>addgroup hadoop</div><div>adduser  hadoop</div><div>usermod -a -G hadoop hadoop</div><div>passwd hadoop</div><div><br></div><div>配置ssh:</div><div><br></div><div>服务端:</div><div>su hadoop</div><div>ssh-keygen  -t  rsa</div><div>cp id_rsa.pub authorized_keys</div><div><br></div><div>客户端</div><div>chmod 700 /home/hadoop</div><div>chmod 755 /home/hadoop/.ssh</div><div>su hadoop</div><div>cd /home</div><div>mkdir .ssh</div><div><br></div><div>服务端:</div><div>chmod 644 /home/hadoop/.ssh/authorized_keys</div><div>scp authorized_keys datanode1:/home/hadoop/.ssh/</div><div>scp authorized_keys datanode2:/home/hadoop/.ssh/</div><div><br></div><div>ssh datanode1</div><div>ssh datanode2</div><div><br></div><div>　如果ssh配置好了就会出现以下提示信息</div><div>The authenticity of host [dbrg-2] can't be established.</div><div>Key fingerpr is 1024 5f:a0:0b:65:d3:82:df:ab:44:62:6d:98:9c:fe:e9:52.</div><div>Are you sure you want to continue connecting (yes/no)?</div><div>　　OpenSSH告诉你它不知道这台主机但是你不用担心这个问题你是第次登录这台主机键入“yes”这将把</div><div>这台主机“识别标记”加到“~/.ssh/know_hosts”文件中第 2次访问这台主机时候就不会再显示这条提示信</div><div><br></div><div><br></div><div>不过别忘了测试本机ssh　dbrg-1</div><div>　</div><div><br></div><div><br></div><div>mkdir /home/hadoop/HadoopInstall</div><div>tar -zxvf hadoop-0.20.1.tar.gz -C /home/hadoop/HadoopInstall/</div><div>cd /home/hadoop/HadoopInstall/</div><div>ln  -s  hadoop-0.20.1  hadoop</div><div><br></div><div>export JAVA_HOME=/usr/local/java</div><div>export CLASSPATH=.:/usr/local/java/lib</div><div>export HADOOP_HOME=/home/hadoop/HadoopInstall/hadoop</div><div>export HADOOP_CONF_DIR=/home/hadoop/hadoop-conf</div><div>export PATH=$HADOOP_HOME/bin:$PATH</div><div><br></div><div>cd $HADOOP_HOME/conf/</div><div>mkdir /home/hadoop/hadoop-conf</div><div>cp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml masters slaves /home/hadoop/hadoop-conf</div><div><br></div><div>vi $HADOOP_HOME/hadoop-conf/hadoop-env.sh</div><div><br></div><div><br></div><div># The java implementation to use.  Required. --修改成你自己jdk安装的目录</div><div>export JAVA_HOME=/usr/local/java </div><div> </div><div>export  HADOOP_CLASSPATH=.:/usr/local/java/lib</div><div># The maximum amount of heap to use, in MB. Default is 1000.--根据你的内存大小调整</div><div>export HADOOP_HEAPSIZE=200          </div><div><br></div><div>vi /home/hadoop/.bashrc</div><div>export JAVA_HOME=/usr/local/java</div><div>export CLASSPATH=.:/usr/local/java/lib</div><div>export HADOOP_HOME=/home/hadoop/HadoopInstall/hadoop</div><div>export HADOOP_CONF_DIR=/home/hadoop/hadoop-conf</div><div>export PATH=$HADOOP_HOME/bin:$PATH</div><div><br></div><div><br></div><div><br></div><div>配置</div><div><br></div><div>namenode</div><div><br></div><div>#vi $HADOOP_CONF_DIR/slaves</div><div>192.168.13.108</div><div>192.168.13.110</div><div><br></div><div>#vi $HADOOP_CONF_DIR/core-site.xml</div><div><?xml version="1.0"?></div><div><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></div><div> </div><div></div><div> </div><div><configuration></div><div><property></div><div>   <name>fs.default.name</name></div><div>   <value>hdfs://192.168.13.100:9000</value></div><div></property></div><div></configuration></div><div><br></div><div>#vi $HADOOP_CONF_DIR/hdfs-site.xml </div><div><?xml version="1.0"?></div><div><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></div><div> </div><div></div><div> </div><div><configuration></div><div><property></div><div>  <name>dfs.replication</name></div><div>  <value>3</value></div><div>  <description>Default block replication.</div><div>  The actual number of replications can be specified when the file is created.</div><div>  The default is used if replication is not specified in create time.</div><div>  </description></div><div></property></div><div></configuration></div><div><br></div><div><br></div><div>#vi $HADOOP_CONF_DIR/mapred-site.xml </div><div><br></div><div><?xml version="1.0"?></div><div><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></div><div> </div><div></div><div> </div><div><configuration></div><div><property></div><div>        <name>mapred.job.tracker</name></div><div>        <value>192.168.13.100:11000</value></div><div>   </property></div><div></configuration></div><div>~                </div><div><br></div><div><br></div><div><br></div><div><br></div><div>在slave上的配置文件如下(hdfs-site.xml不需要配置)：</div><div>[root@test12 conf]# cat core-site.xml </div><div><?xml version="1.0"?></div><div><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></div><div><br></div><div></div><div><br></div><div><configuration></div><div>   <property></div><div>      <name>fs.default.name</name></div><div>      <value>hdfs://namenode:9000</value></div><div>   </property></div><div></configuration></div><div><br></div><div>[root@test12 conf]# cat mapred-site.xml </div><div><?xml version="1.0"?></div><div><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></div><div><br></div><div></div><div><br></div><div><configuration></div><div>   <property></div><div>        <name>mapred.job.tracker</name></div><div>        <value>namenode:11000</value></div><div>   </property></div><div></configuration></div><div><br></div><div><br></div><div><br></div><div>启动</div><div>export PATH=$HADOOP_HOME/bin:$PATH</div><div><br></div><div>hadoop namenode -format</div><div>start-all.sh</div><div>停止stop-all.sh</div><div><br></div><div>在hdfs上创建danchentest文件夹，上传文件到此目录下</div><div>$HADOOP_HOME/bin/hadoop fs -mkdir danchentest</div><div>$HADOOP_HOME/bin/hadoop fs -put $HADOOP_HOME/README.txt danchentest</div><div><br></div><div>cd $HADOOP_HOME</div><div>hadoop jar hadoop-0.20.1-examples.jar wordcount  /user/hadoop/danchentest/README.txt output1</div><div>09/12/21 18:31:44 INFO input.FileInputFormat: Total input paths to process : 1</div><div>09/12/21 18:31:45 INFO mapred.JobClient: Running job: job_200912211824_0002</div><div>09/12/21 18:31:46 INFO mapred.JobClient:  map 0% reduce 0%</div><div>09/12/21 18:31:53 INFO mapred.JobClient:  map 100% reduce 0%</div><div>09/12/21 18:32:05 INFO mapred.JobClient:  map 100% reduce 100%</div><div>09/12/21 18:32:07 INFO mapred.JobClient: Job complete: job_200912211824_0002</div><div>09/12/21 18:32:07 INFO mapred.JobClient: Counters: 17</div><div>09/12/21 18:32:07 INFO mapred.JobClient:   Job Counters </div><div>09/12/21 18:32:07 INFO mapred.JobClient:     Launched reduce tasks=1</div><div><br></div><div>查看输出结果文件，这个文件在hdfs上</div><div>[root@test11 hadoop]# hadoop fs -ls output1</div><div>Found 2 items</div><div>drwxr-xr-x   - root supergroup          0 2009-09-30 16:01 /user/root/output1/_logs</div><div>-rw-r--r--   3 root supergroup       1306 2009-09-30 16:01 /user/root/output1/part-r-00000</div><div><br></div><div>[root@test11 hadoop]# hadoop fs -cat output1/part-r-00000</div><div>(BIS),  1</div><div>(ECCN)  1</div><div><br></div><div>查看hdfs运行状态,可以通过web界面来访问http://192.168.13.100:50070/dfshealth.jsp;查看map-reduce信息，</div><div>可以通过web界面来访问http://192.168.13.100:50030/jobtracker.jsp;下面是直接命令行看到的结果。</div><div><br></div><div><br></div><div>出现08/01/25 16:31:40 INFO ipc.Client: Retrying connect to server: foo.bar.com/1.1.1.1:53567. Already tried 1 time(s).</div><div>的原因是没有格式化:hadoop namenode -format</div>

文库|博客

返回列表

Chinaunix › 论坛 › 数据库技术 › NoSQL技术 › linux下安装hadoop步骤

[Hadoop&HBase] linux下安装hadoop步骤 [复制链接]