论坛徽章:: 0

电梯直达

1楼 [收藏(0)] [报告]

发表于 2009-09-27 11:20 |只看该作者 |倒序浏览

【原创】基于FreeBSD的分布式文件系统hadoop(手记)前言，经过了将近一个星期的摸索，由于之前用过freebsd配置过apache+mysql+zend+php环境，对freebsd的操作还算熟悉，又从网上搜集了一些资料进行配置，目前能正常运行，仅仅是试用，暂时还没有投入系统中用，正在解决api问题，下面步骤可能有错，希望高手指正~~ 欢迎转帖，转帖请注明原作者，pigwyl QQ：86206221 email：pigwyl#gmail.com 非常感谢学校的网络中心的于老师 论坛id:flyfish 给我提供了配置的环境 一，准备 1.安装软件版本 freebsd: 7.2 jdk1.5 2.安装系统 用完全安装否则jdk有可能安装失败 SSH得安装与配置 首先ee编辑/etc/inetd.conf,去掉ssh前的#，保存退出 编辑/etc/rc.conf 最后加入:sshd_enable="yes"即可 激活sshd服务： #/etc/rc.d/sshd start 最后 ee /etc/ssh/sshd_config, 下面是我的配置文件：(/etc/ssh/sshd_config) ####################################################

rotocol 2 AllowGroups wheel#允许wheel组成员 IgnoreRhosts yes IgnoreUserKnownHosts yes

rintMotd yes StrictModes no RSAAuthentication yes X11Forwarding no

ermitRootLogin yes #允许root登录

ermitEmptyPasswords yes #允许空密码登录

asswordAuthentication yes # 设置是否使用口令验证。 MaxStartups 5 AuthorizedKeysFile .ssh/authorized_keys ############################################## 记得修改完配置文件后，重新启动sshd服务器(/etc/rc.d/sshd restart)即可。 添加一个用户 #sysinstall 选择【Configure】--【User Management】--【User】，只需member group 填wheel即可，其他按自己要求填 重启机器后即可用SecureCRT了。 二，os及环境配置 1，安装jdk1.5 首先下载 http://karakurty.info/ftp/pub/FreeBSD/ports/distfiles/jdk-1_5_0_16-fcs-bin-b02-jrl-28_may_2008.jar http://karakurty.info/ftp/pub/FreeBSD/ports/distfiles/jdk-1_5_0_16-fcs-src-b02-jrl-28_may_2008.jar http://karakurty.info/ftp/pub/FreeBSD/ports/distfiles/tzupdater-1_3_12-2009a.zip http://karakurty.info/ftp/pub/FreeBSD/ports/distfiles/bsd-jdk15-patches-9.tar.bz2 http://www.freebsdfoundation.org/cgi-bin/download?download=diablo-caffe-freebsd7-i386-1.6.0_07-b02.tar.bz2 到usr/ports/distfiles/目录 #cd /usr/ports/java/jdk15 # make install clean 编译完成之后会 ===>Cleaning for unzip-5.52_5 ===>Cleaning for m4-1.4.12,1 ===>Cleaning for zip-3.0 ===>Cleaning for open-motif-2.2.3_6 ===>Cleaning for gmake-3.81_3 ===>Cleaning for libX11-1.2.1,1 ===>Cleaning for libXext-1.0.5,1 ===>Cleaning for libXi-1.2.1,1 ===>Cleaning for libXmu-1.0.4,1 ===>Cleaning for libXp-1.0.0,1 ===>Cleaning for libXt-1.0.5_1 ===>Cleaning for libXtst-1.0.3_1 ===>Cleaning for pkg-config-0.23_1 ===>Cleaning for desktop-file-utils-0.15_1 ===>Cleaning for nspr-4.7 ===>Cleaning for libiconv-1.11_1 ===>Cleaning for glib-2.20.1 ===>Cleaning for javavmwrapper-2.3.2 ===>Cleaning for gio-fam-backend-2.20.1 ===>Cleaning for libsigsegv-2.5 ===>Cleaning for libXaw-1.0.5_1,1 ===>Cleaning for xbitmaps-1.0.1 ===>Cleaning for libtool-1.5.26 ===>Cleaning for gettext-0.17_1 ===>Cleaning for libxcb-1.2_1 ===>Cleaning for xorg-macros-1.2.1 ===>Cleaning for bigreqsproto-1.0.2 ===>Cleaning for xcmiscproto-1.1.2 ===>Cleaning for xextproto-7.0.5 ===>Cleaning for xtrans-1.2.3 ===>Cleaning for kbproto-1.0.3 ===>Cleaning for inputproto-1.5.0 ===>Cleaning for xf86bigfontproto-1.1.2 ===>Cleaning for libXau-1.0.4 ===>Cleaning for libXdmcp-1.0.2_1 ===>Cleaning for xproto-7.0.15 ===>Cleaning for automake-1.10.1 ===>Cleaning for autoconf-2.62 ===>Cleaning for printproto-1.0.4 ===>Cleaning for libSM-1.1.0_1,1 ===>Cleaning for recordproto-1.13.2 ===>Cleaning for perl-5.8.9_2 ===>Cleaning for python25-2.5.4_1 ===>Cleaning for pcre-7.9 ===>Cleaning for gamin-0.1.10_1 ===>Cleaning for libXpm-3.5.7 ===>Cleaning for libcheck-0.9.6 ===>Cleaning for libxslt-1.1.24_2 ===>Cleaning for xcb-proto-1.4 ===>Cleaning for libpthread-stubs-0.1 ===>Cleaning for automake-wrapper-20071109 ===>Cleaning for help2man-1.36.4_2 ===>Cleaning for autoconf-wrapper-20071109 ===>Cleaning for libICE-1.0.4_1,1 ===>Cleaning for libxml2-2.7.3 ===>Cleaning for p5-gettext-1.05_2 ===>Cleaning for jdk-1.5.0.16p9_1,1 2，安装rsync，bash a, #cd /usr/ports/net/rsync #make install clean b, #cd /usr/ports/shells/bash #make install clean 3，系统配置 a, 本次配置使用服务器分别为 vc1(用在namenode由于本次测试只有2台机器所以兼datanode，但实际生产环境中不推荐兼datanode) vc2(用在datanode) b,配置服务器的hosts文件使namenode能够通过域名(vc1和vc2）访问到所有的datanode（如果namenode兼datanode也需要通过域名访问到自己）， 所有的datanode也能通过域名访问到namenode c，示例 ==========================/etc/hosts=========================================== ::1 localhost localhost.rhinux.com 127.0.0.1 localhost localhost.rhinux.com 200.0.0.003 vc1.tj.com vc1 200.0.0.004 vc2.tj.com vc2 ======================（namenode & datanode）================================== 免密码ssh设置 如果不输入口令就无法用ssh登陆localhost，执行下面的命令： 1 生成密钥对,简便起见，在vc2上来生成,一下操作我用的都是hadoop ssh-keygen -t rsa ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_dsa. Your public key has been saved in /root/.ssh/id_dsa.pub. The key fingerprint is: c3<img src="http://bbs.chinacloud.cn/editor/images/smilies/default/35.gif" border="0" onclick="zoom(this, this.src)" onload="attachimg(this, 'load')" alt="" />4:4d:24:64<img src="http://bbs.chinaitlab.com/images/smilies/mogutou/thumb_05we03288.gif" smilieid="47" border="0" alt="" />4:9d:2b:76:1a:6f:7e<img src="http://bbs.chinacloud.cn/editor/images/smilies/default/22.gif" border="0" onclick="zoom(this, this.src)" onload="attachimg(this, 'load')" alt="" />c:1a:60:23 root@fedora3 其中id_dsa是私钥，id_dsa.pub是公钥，私钥保留在客户端本地，公钥要复制到你要ssh登录的服务器上 2 配置公钥 在vc1上复制公钥到服务器vc2 scp ~/.ssh/id_dsa.pub vc2:/tmp ssh a cat /tmp/id_dsa.pub >> ~/.ssh/authorized_keys rm -rf /tmp/id_rsa.pub 然后下载hadoop-0.18.3.tar.gz 解压到/usr/home/hadoop/目录 编辑/usr/home/hadoop/hadoop-0.18.3/conf/hadoop-env.sh export JAVA_HOME=/usr/local/jdk1.5.0#更改java的路径 export HADOOP_HOME=/usr/home/pig/hadoop-0.18.3 #hadoop的路径 export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS" export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS" export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS" export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS" 回到/usr/home/hadoop/hadoop-0.18.3目录 #cd bin #./start-all.sh 编辑 conf/master localhost conf/slaves localhost vc1 vc2 配置conf/hadoop-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  <configuration> <property> <name>fs.default.name</name>//你的namenode的配置，机器名加端口 <value>hdfs://200.0.0.003:54310/</value> </property> <property> <name>mapred.job.tracker</name>//你的JobTracker的配置，机器名加端口 <value>hdfs://200.0.0.003:54311/</value> </property> <property> <name>dfs.replication</name>//数据需要备份的数量，默认是三 <value>1</value> </property> <property> <name>hadoop.tmp.dir</name>//Hadoop的默认临时路径，这个最好配置，如果在新增节点或者其他情况下莫名其妙的DataNode启动不了，就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录，那么就需要重新执行NameNode格式化的命令。 <value>/home/wenchu/hadoop/tmp/</value> </property> <property> <name>mapred.child.java.opts</name>//java虚拟机的一些参数可以参照配置 <value>-Xmx512m</value> </property> <property> <name>dfs.block.size</name>//block的大小，单位字节，后面会提到用处，必须是512的倍数，因为采用crc作文件完整性校验，默认配置512是checksum的最小单元。 <value>5120000</value> <description>The default block size for new files.</description> </property> </configuration> 然后启动hadoop守护进程， #cd /usr/hadoop/hadoop-0.18.3/bin #./start-all.sh 然后查看一下是否正常运行 #./hadoop dfsadmin -report