- 论坛徽章:
- 0
|
<A href="http://blog.chinaunix.nethttp://blog.chinaunix.net/attachment/201101/19/25098298_1295430090KP3p.jpg" target=_blank></A>1.环境准备<BR>jdk-6u20-linux-i586.bin（据称jdk在6u20之后的版本，有包名修改，与eclipse3.6集成安装会出现不兼容，所以在此采用jdk6u20版本）<BR>hadoop-0.21.0.tar.gz<BR>ubuntu10.10<BR><BR>2. 安装 jdk-6u20-linux-i586.bin<BR><BR>复制jdk-6u20-linux-i586.bin 安装文件到/usr目录下<BR>sudo cp /home/chensl/software/jdk-6u20-linux-i586.bin /usr<BR><BR> 进入/usr目录，,先设置bin文件可执行，执行安装文件<BR>sudo chmod a+x jdk-6u20-linux-i586.bin<BR>sudo ./jdk-6u20-linux-i586.bin <BR><BR>配置JAVA环境变量<BR>运行命令<BR>sudo gedit /etc/environment<BR>在首部添加<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>JAVA_HOME="/usr/jdk1.6.0_20"</LI>
<LI>CLASSPATH=".:/usr/jdk1.6.0_20/lib"</LI></OL></DIV>并修改PATH，后面添加JAVA_HOME/bin部分<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:$JAVA_HOME/bin"</LI></OL></DIV>运行 java -version,会有下面的提示：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>程序“java”已包含在下列软件包中：</LI>
<LI>* gcj-4.4-jre-headless</LI>
<LI>* gcj-4.5-jre-headless</LI>
<LI>* openjdk-6-jre-headless</LI>
<LI>请尝试：sudo apt-get install <选定的软件包></LI></OL></DIV>执行下面两个命令，安装java到系统中<SPAN style="FONT-FAMILY: sans-serif"><BR></SPAN><SPAN style="FONT-FAMILY: sans-serif">sudo update-alternatives --install /usr/bin/java java /usr/jdk1.6.0_20/bin/java 300<BR></SPAN><BR>sudo update-alternatives --install /usr/bin/javac javac /usr/jdk1.6.0_20/bin/javac 300<BR>
<P></P>设置默认的JDK<BR>sudo update-alternatives --config java<BR><BR>查看java版本号：<BR>java -version<BR>结果输出：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>java version "1.6.0_20"</LI>
<LI>Java(TM) SE Runtime Environment (build 1.6.0_20-b02)</LI>
<LI>Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)</LI></OL></DIV>jdk配置完成<BR><BR>3.配置SSH<BR>如果没有安装SSH，需要先安装配置，sudo apt-get install openssh-server （<A href="http://blog.chinaunix.net/space.php?uid=25098298&do=blog&id=79870" target=_blank>具体可参见</A>）<BR><BR>（1）快速配置方式，设定无需登录密码方式：<BR> 由于Hadoop用ssh做机器间的沟通，因此可以设定登录机器免密码<BR>ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ""<BR>cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys<BR>测试下，第一次登录需要yes下，第二次直接进入<BR>ssh localhost<BR><BR>（2）添加一个用户的配置方式<BR>方便起见，新增hadoop的组及其同名用户：<BR>sudo addgroup hadoop<BR>sudo adduser --ingroup hadoop hadoop<BR><BR>此时如果使用hadoop用户，使用sudo命令时，会提示下面的内容：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>[sudo] password for hadoop: </LI>
<LI>hadoop is not in the sudoers file. This incident will be reported.</LI></OL></DIV>表明hadoop用户还无法运行sudo的命令 ，接下来需要做下特殊的工作：<BR>su （转换成root用户）<BR>ls -l /etc/sudoers 显示如下：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>-r--r----- 1 root root 567 2011-01-09 10:30 /etc/sudoers</LI></OL></DIV>chmod u+w /etc/sudoers （修改属性）<BR>ls -l /etc/sudoers 这次显示如下：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>-rw-r----- 1 root root 567 2011-01-09 10:30 /etc/sudoers</LI></OL></DIV>gedit /etc/suders （编辑suders文件）<BR>在 root ALL=(ALL) ALL 后面添加<BR>hadoop ALL=(ALL) ALL<BR><BR>chmod u-w /etc/sudoers （修改回原来的属性）<BR>exit （ 退出root管理员）<BR><BR>如果还没安装openssh-server的话，运行 sudo apt-get install openssh-server 进行安装<BR>下面建立SSH KEY：<BR>su hadoop （使用hadoop用户）<BR>ssh-keygen -t rsa -P ""<BR>界面输出：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>Generating public/private rsa key pair.</LI>
<LI>Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): </LI>
<LI>Created directory '/home/hadoop/.ssh'.</LI>
<LI>Your identification has been saved in /home/hadoop/.ssh/id_rsa.</LI>
<LI>Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.</LI>
<LI>The key fingerprint is:</LI>
<LI>3f:a7:74:82:5c:8a:68:16:ff:21:d5:90:08:9d:fa:9f hadoop@chensl-virtual-machine</LI>
<LI>The key's randomart image is:</LI>
<LI>+--[ RSA 2048]----+</LI>
<LI>| .. . |</LI>
<LI>| .o. . |</LI>
<LI>| .. o |</LI>
<LI>| . o |</LI>
<LI>| .. S o |</LI>
<LI>| +.+ = |</LI>
<LI>| + +.=.= o |</LI>
<LI>| o oEo * |</LI>
<LI>| . . |</LI>
<LI>+-----------------+</LI></OL></DIV>启用SSH KEY：<BR>cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized.keys<BR>运行命令：<BR>sudo /etc/init.d/ssh reload<BR>验证登录命令：<BR>ssh localhost<BR><BR>4. 安装配置hadoop<BR>（1）下载及安装<BR>前面的环境准备工作完成后，现在就可以安装并配置hadoop了<BR>下载hadoop-0.21.0.tar.gz 版本，并解压到/usr/local/hadoop下，并修改所有者为hadoop<BR>在/usr/local/目录下执行：<BR>sudo tar -zxvf hadoop-0.21.0.tar.gz<BR>重命名文件夹 <BR>sudo mv hadoop-0.21.0 hadoop<BR>修改所有者为hadoop<BR>sudo chown -R hadoop:hadoop hadoop<BR><BR>（2）配置：<BR>配置 $HADOOP_HOME/conf/hadoop-env.sh<BR>使用hadoop用户，进入/usr/local/hadoop 目录：<BR>然后运行命令：<BR>sudo gedit conf/hadoop-env.sh<BR><BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>将<BR></LI>
<LI># export JAVA_HOME=/usr/lib/j2sdk1.6-sun</LI>
<LI>修改为：</LI>
<LI>export JAVA_HOME=/usr/jdk1.6.0_20 （java的安装目录）</LI></OL></DIV>配置：conf/core-site.xml<BR>
<DIV class=codeText id=codeText style="WIDTH: 98%; HEIGHT: 120px">
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px"><configuration><BR> <property><BR> <name>fs.default.name</name><BR> <value>hdfs://localhost:9000</value><BR> </property><BR></configuration></OL></DIV>注意：如果没有配置hadoop.tmp.dir目录，那么系统的默认的临时目录为/tmp/hadoop-hadoop 这个目录每次在重启后都会被干掉，必须重新执行format才行，否则出错。<BR>修改conf/hdfs-site.xml ：<BR>
<DIV class=codeText id=codeText style="WIDTH: 98%; HEIGHT: 122px">
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px"><configuration><BR> <property><BR> <name>dfs.replication</name><BR> <value>1</value><BR> </property><BR></configuration></OL></DIV>修改conf/mapred-site.xml：<BR>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px"><configuration><BR> <property><BR> <name>mapred.job.tracker</name><BR> <value>localhost:9001</value><BR> </property><BR></configuration><BR></OL></DIV>
<P>格式化namenode<BR>./bin/hadoop namenode -format</P>
<P>启动hadoop</P>
<P>./bin/start-all.sh 将依次启动namenode，datanode,secondarynamenode,jobtracker等几项，并要求输入hadoop用户密码。</P>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0px; PADDING-LEFT: 0px; PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-TOP: 5px">
<LI>This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh</LI>
<LI><BR></LI>
<LI>starting namenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-namenode-chensl-virtual-machine.out</LI>
<LI><BR></LI>
<LI>hadoop@localhost's password: </LI>
<LI><BR></LI>
<LI>localhost: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-datanode-chensl-virtual-machine.out</LI>
<LI><BR></LI>
<LI>hadoop@localhost's password: </LI>
<LI><BR></LI>
<LI>localhost: starting secondarynamenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-chensl-virtual-machine.out</LI>
<LI><BR></LI>
<LI>starting jobtracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-jobtracker-chensl-virtual-machine.out</LI>
<LI><BR></LI>
<LI>hadoop@localhost's password: </LI>
<LI><BR></LI>
<LI>localhost: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hadoop-tasktracker-chensl-virtual-machine.out</LI></OL></DIV>
<P>验证方法一：</P>
<P>在浏览器中输入下面两个地址。可以分别查看NameNode和JobTracker</P>
<UL>
<LI><SPAN class=codefrag>NameNode</SPAN> - <A href="http://localhost:50070/" target=_blank>http://localhost:50070/</A> </LI>
<LI><SPAN class=codefrag>JobTracker</SPAN> - <A href="http://localhost:50030/" target=_blank>http://localhost:50030/</A> </LI></UL>
<P>其中： <A href="http://localhost:50070/" target=_blank>http://localhost:50070/</A> 的截图为：</P>
<P><A href="http://blog.chinaunix.nethttp://blog.chinaunix.net/attachment/201101/19/25098298_1295430865J2Z2.jpg" target=_blank><IMG src="http://blog.chinaunix.nethttp://blog.chinaunix.net/attachment/201101/19/25098298_1295430865J2Z2.jpg" border=0 ;="" .load="imgResize(this, 650);"></A></P>
<P><A href="http://localhost:50030/" target=_blank>http://localhost:50030/</A> 的截图为：</P>
<P><A href="http://blog.chinaunix.nethttp://blog.chinaunix.net/attachment/201101/19/25098298_1295430090KP3p.jpg" target=_blank><IMG src="http://blog.chinaunix.nethttp://blog.chinaunix.net/attachment/201101/19/25098298_1295430090KP3p.jpg" border=0 ;="" .load="imgResize(this, 650);"></A><BR><BR>配置运行hadoop成功。</P>
<P>验证方法二：运行jps</P>
<P>/usr/jdk1.6.0_20/bin/jps 显示如下内容： <BR></P>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0px; PADDING-LEFT: 0px; PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-TOP: 5px">
<LI>7768 TaskTracker</LI>
<LI>7559 JobTracker</LI>
<LI>7243 DataNode</LI>
<LI>7972 Jps</LI>
<LI>7462 SecondaryNameNode</LI>
<LI>7039 NameNode</LI></OL></DIV>
<P>验证方法三：</P>
<P>运行如下命令：</P>
<P>bin/hadoop dfsadmin -report 界面显示：</P>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>DEPRECATED: Use of this script to execute hdfs command is deprecated.</LI>
<LI>Instead use the hdfs command for it.</LI>
<LI><BR></LI>
<LI>11/01/19 17:58:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000</LI>
<LI>11/01/19 17:58:25 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id</LI>
<LI>Configured Capacity: 40493293568 (37.71 GB)</LI>
<LI>Present Capacity: 34237849600 (31.89 GB)</LI>
<LI>DFS Remaining: 34237808640 (31.89 GB)</LI>
<LI>DFS Used: 40960 (40 KB)</LI>
<LI>DFS Used%: 0%</LI>
<LI>Under replicated blocks: 0</LI>
<LI>Blocks with corrupt replicas: 0</LI>
<LI>Missing blocks: 0</LI>
<LI><BR></LI>
<LI>-------------------------------------------------</LI>
<LI>Datanodes available: 1 (1 total, 0 dead)</LI>
<LI><BR></LI>
<LI>Live datanodes:</LI>
<LI>Name: 127.0.0.1:50010 (localhost.localdomain)</LI>
<LI>Decommission Status : Normal</LI>
<LI>Configured Capacity: 40493293568 (37.71 GB)</LI>
<LI>DFS Used: 40960 (40 KB)</LI>
<LI>Non DFS Used: 6255443968 (5.83 GB)</LI>
<LI>DFS Remaining: 34237808640 (31.89 GB)</LI>
<LI>DFS Used%: 0%</LI>
<LI>DFS Remaining%: 84.55%</LI>
<LI>Last contact: Wed Jan 19 17:58:22 CST 2011</LI></OL></DIV>
<P> 以上表示hadoop配置并启动正常，另外也可以查看$HADOOP_HOME/logs下的日志信息。</P>
<P>根据apache官方文档，可以执行命令，实验下面步骤：</P>
<P>在$HADOOP_HOME路径下：</P>
<P>mkdir input</P>
<P>cp conf/*.xml input</P>
<P>bin/hadoop fs -put input/ input （复制input目录下的文件到分布式系统input目录中）</P>
<P>bin/hadoop jar hadoop-*-examples*.jar grep input output 'dfs[a-z.]+' （执行例子程序）</P>
<P>输出下面内容：</P>
<DIV class=codeText id=codeText>
<OL class=dp-css style="PADDING-RIGHT: 0pt; PADDING-LEFT: 0pt; PADDING-BOTTOM: 5px; MARGIN: 0pt 1px 0pt 0pt; PADDING-TOP: 5px">
<LI>11/01/19 18:44:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000</LI>
<LI>11/01/19 18:44:26 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id</LI>
<LI>11/01/19 18:44:26 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.</LI>
<LI>11/01/19 18:44:26 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).</LI>
<LI>11/01/19 18:44:26 INFO input.FileInputFormat: Total input paths to process : 7</LI>
<LI>11/01/19 18:44:26 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps</LI>
<LI>11/01/19 18:44:26 INFO mapreduce.JobSubmitter: number of splits:7</LI>
<LI>11/01/19 18:44:26 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null</LI>
<LI>11/01/19 18:44:26 INFO mapreduce.Job: Running job: job_201101191734_0010</LI>
<LI>11/01/19 18:44:27 INFO mapreduce.Job: map 0% reduce 0%</LI>
<LI>11/01/19 18:44:39 INFO mapreduce.Job: map 28% reduce 0%</LI>
<LI>11/01/19 18:44:48 INFO mapreduce.Job: map 57% reduce 9%</LI>
<LI>11/01/19 18:44:54 INFO mapreduce.Job: map 71% reduce 19%</LI>
<LI>11/01/19 18:45:00 INFO mapreduce.Job: map 85% reduce 19%</LI>
<LI>11/01/19 18:45:03 INFO mapreduce.Job: map 100% reduce 23%</LI>
<LI>11/01/19 18:45:06 INFO mapreduce.Job: map 100% reduce 28%</LI>
<LI>11/01/19 18:45:12 INFO mapreduce.Job: map 100% reduce 100%</LI>
<LI>11/01/19 18:45:14 INFO mapreduce.Job: Job complete: job_201101191734_0010</LI>
<LI>11/01/19 18:45:14 INFO mapreduce.Job: Counters: 33</LI>
<LI>FileInputFormatCounters</LI>
<LI>BYTES_READ=12244</LI>
<LI>FileSystemCounters</LI>
<LI>FILE_BYTES_READ=51</LI>
<LI>FILE_BYTES_WRITTEN=362</LI>
<LI>HDFS_BYTES_READ=13094</LI>
<LI>HDFS_BYTES_WRITTEN=143</LI>
<LI>Shuffle Errors</LI>
<LI>BAD_ID=0</LI>
<LI>CONNECTION=0</LI>
<LI>IO_ERROR=0</LI>
<LI>WRONG_LENGTH=0</LI>
<LI>WRONG_MAP=0</LI>
<LI>WRONG_REDUCE=0</LI>
<LI>Job Counters </LI>
<LI>Data-local map tasks=7</LI>
<LI>Total time spent by all maps waiting after reserving slots (ms)=0</LI>
<LI>Total time spent by all reduces waiting after reserving slots (ms)=0</LI>
<LI>SLOTS_MILLIS_MAPS=53950</LI>
<LI>SLOTS_MILLIS_REDUCES=29573</LI>
<LI>Launched map tasks=7</LI>
<LI>Launched reduce tasks=1</LI>
<LI>Map-Reduce Framework</LI>
<LI>Combine input records=2</LI>
<LI>Combine output records=2</LI>
<LI>Failed Shuffles=0</LI>
<LI>GC time elapsed (ms)=754</LI>
<LI>Map input records=305</LI>
<LI>Map output bytes=41</LI>
<LI>Map output records=2</LI>
<LI>Merged Map outputs=7</LI>
<LI>Reduce input groups=2</LI>
<LI>Reduce input records=2</LI>
<LI>Reduce output records=2</LI>
<LI>Reduce shuffle bytes=87</LI>
<LI>Shuffled Maps =7</LI>
<LI>Spilled Records=4</LI>
<LI>SPLIT_RAW_BYTES=850</LI>
<LI>11/01/19 18:45:14 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.</LI>
<LI>11/01/19 18:45:14 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).</LI>
<LI>11/01/19 18:45:14 INFO input.FileInputFormat: Total input paths to process : 1</LI>
<LI>11/01/19 18:45:14 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps</LI>
<LI>11/01/19 18:45:14 INFO mapreduce.JobSubmitter: number of splits:1</LI>
<LI>11/01/19 18:45:14 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null</LI>
<LI>11/01/19 18:45:14 INFO mapreduce.Job: Running job: job_201101191734_0011</LI>
<LI>11/01/19 18:45:15 INFO mapreduce.Job: map 0% reduce 0%</LI>
<LI>11/01/19 18:45:24 INFO mapreduce.Job: map 100% reduce 0%</LI>
<LI>11/01/19 18:45:30 INFO mapreduce.Job: map 100% reduce 100%</LI>
<LI>11/01/19 18:45:32 INFO mapreduce.Job: Job complete: job_201101191734_0011</LI>
<LI>11/01/19 18:45:32 INFO mapreduce.Job: Counters: 33</LI>
<LI>FileInputFormatCounters</LI>
<LI>BYTES_READ=143</LI>
<LI>FileSystemCounters</LI>
<LI>FILE_BYTES_READ=51</LI>
<LI>FILE_BYTES_WRITTEN=134</LI>
<LI>HDFS_BYTES_READ=274</LI>
<LI>HDFS_BYTES_WRITTEN=29</LI>
<LI>Shuffle Errors</LI>
<LI>BAD_ID=0</LI>
<LI>CONNECTION=0</LI>
<LI>IO_ERROR=0</LI>
<LI>WRONG_LENGTH=0</LI>
<LI>WRONG_MAP=0</LI>
<LI>WRONG_REDUCE=0</LI>
<LI>Job Counters </LI>
<LI>Data-local map tasks=1</LI>
<LI>Total time spent by all maps waiting after reserving slots (ms)=0</LI>
<LI>Total time spent by all reduces waiting after reserving slots (ms)=0</LI>
<LI>SLOTS_MILLIS_MAPS=4761</LI>
<LI>SLOTS_MILLIS_REDUCES=3152</LI>
<LI>Launched map tasks=1</LI>
<LI>Launched reduce tasks=1</LI>
<LI>Map-Reduce Framework</LI>
<LI>Combine input records=0</LI>
<LI>Combine output records=0</LI>
<LI>Failed Shuffles=0</LI>
<LI>GC time elapsed (ms)=93</LI>
<LI>Map input records=2</LI>
<LI>Map output bytes=41</LI>
<LI>Map output records=2</LI>
<LI>Merged Map outputs=1</LI>
<LI>Reduce input groups=1</LI>
<LI>Reduce input records=2</LI>
<LI>Reduce output records=2</LI>
<LI>Reduce shuffle bytes=51</LI>
<LI>Shuffled Maps =1</LI>
<LI>Spilled Records=4</LI>
<LI>SPLIT_RAW_BYTES=131</LI></OL></DIV>
<P>检查运行结果：</P>
<P>bin/hadoop fs -get output output （从分布式系统中复制output目录到本地系统）</P>
<P>cat output/*</P>
<P> 或者使用命令直接查看：</P>
<P>bin/hadoop fs -cat output/*</P>运行完以上示例，可以用下面的命令停止hadoop
<P>bin/stop-all.sh</P>
<P>(注：其他可能会用到的命令如：bin/hadoop fs -rmr input 表示删除分布式系统中的input目录)</P><BR>
<P>参考：</P>
<P><a href="http://itstarting.javaeye.com/blog/520985" target="_blank">http://itstarting.javaeye.com/blog/520985</A> <BR>http://hadoop.apache.org/common/docs/r0.21.0/single_node_setup.html<BR>http://wenku.baidu.com/view/55a4163f5727a5e9856a61ed.html<BR>http://wenku.baidu.com/view/453a23fff705cc17552709ee.html<BR></P> |
|
免费注册
发表于 2011-12-19 13:54