免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
查看: 988 | 回复: 0

网络工程--吴鸣悦组 第二次报告 [复制链接]

论坛徽章:
0
发表于 2011-12-23 02:39 |显示全部楼层
<p align="center"><font size="4">第二次报告</font></p>
<p align="left"><font size="4">组长:吴鸣悦</font></p>
<p align="left"><font size="4">组员:郭宇,赵新蕾,吴宇航,顾思嘉</font></p>
<p style="TEXT-ALIGN: left; MARGIN: 0cm 0cm 0pt; mso-pagination: widow-orphan" class="MsoNormal" align="left"><span style="FONT-FAMILY: 宋体; FONT-SIZE: 12pt; mso-bidi-font-family: 宋体; mso-font-kerning: 0pt" lang="EN-US">HBase</span><span style="FONT-FAMILY: 宋体; FONT-SIZE: 12pt; mso-bidi-font-family: 宋体; mso-font-kerning: 0pt">的安装、配置、管理与编程<span lang="EN-US"> </span></span></p>
<p align="left">
<table style="WIDTH: 100%; mso-cellspacing: 1.5pt" class="MsoNormalTable" border="0" cellpadding="0" width="100%">
<tbody>
<tr style="mso-yfti-irow: 0; mso-yfti-firstrow: yes; mso-yfti-lastrow: yes">
<td style="BORDER-BOTTOM: #ece9d8; BORDER-LEFT: #ece9d8; PADDING-BOTTOM: 0.75pt; BACKGROUND-COLOR: transparent; PADDING-LEFT: 0.75pt; PADDING-RIGHT: 0.75pt; BORDER-TOP: #ece9d8; BORDER-RIGHT: #ece9d8; PADDING-TOP: 0.75pt">
<p style="TEXT-ALIGN: left; MARGIN: 0cm 0cm 0pt; mso-pagination: widow-orphan" class="MsoNormal" align="left"><span style="FONT-FAMILY: 宋体; FONT-SIZE: 12pt; mso-bidi-font-family: 宋体; mso-font-kerning: 0pt">需要环境:<span lang="EN-US"><br><br>PC-1&nbsp; Suse Linux 9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10.192.1.1<br><br>PC-2&nbsp; Suse Linux 9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10.192.1.2<br><br>PC-3&nbsp; Suse Linux 9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10.192.1.3<br><br>PC-4&nbsp; Suse Linux 9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 10.192.1.4<br><br></span>其中,<span lang="EN-US">PC-1</span>做<span lang="EN-US">namenode</span>节点,<span lang="EN-US">PC-2</span>、<span lang="EN-US">PC-3</span>和<span lang="EN-US">PC-4</span>做<span lang="EN-US">datanode</span>节点。<span lang="EN-US"><br><br></span>并且已经安装成功<span lang="EN-US">Hadoop-0.20.1</span>及以上版本。<span lang="EN-US"><br></span>安装包准备<span lang="EN-US"><br><br></span>需要安装包:<span lang="EN-US"><br><br>zookeeper-3.2.1.tar.gz</span>(<span lang="EN-US">stable</span>版本)<span lang="EN-US"><br><br>hbase-0.20.1.tar.gz</span>(<span lang="EN-US">stable</span>版本)<span lang="EN-US"><br></span>安装步骤<span lang="EN-US"><br><br></span>安装和配置<span lang="EN-US">ZooKeeper<br><br>HBase</span>从<span lang="EN-US">0.20.0</span>开始,需要首先安装<span lang="EN-US">ZooKeeper</span>。从<span lang="EN-US">apache</span>上下载<span lang="EN-US">zookeeper-3.2.1.tar.gz</span>(<span lang="EN-US">Stable</span>版本),解压到<span lang="EN-US">/home/hdfs/</span>目录下。<span lang="EN-US"><br><br></span>(<span lang="EN-US">1</span>),在<span lang="EN-US">namenode</span>节点新建<span lang="EN-US">zookeeper</span>目录,在该目录下新建<span lang="EN-US">myid</span>文件。<span lang="EN-US"><br><br></span>(<span lang="EN-US">2</span>),在<span lang="EN-US">zookeeper-3.2.1/conf</span>目录下,拷贝<span lang="EN-US">zoo_sample.cfg</span>为<span lang="EN-US">zoo.cfg</span>。在<span lang="EN-US">zoo.cfg</span>中将<span lang="EN-US">dataDir</span>改为<span lang="EN-US">/home/hdfs/zookeeper</span>,在文件末位添加所有的主机:<span lang="EN-US"><br><br>server.1=10.192.1.1:2888:3888<br><br>server.2=10.192.1.2:2888:3888<br><br>server.3=10.192.1.3:2888:3888<br><br>server.4=10.192.1.4:2888:3888<br><br>server.5=10.192.1.5:2888:3888<br><br>server.6=10.192.1.62888:3888<br><br></span>(<span lang="EN-US">3</span>)用<span lang="EN-US">scp</span>命令将<span lang="EN-US">namenode</span>节点的的<span lang="EN-US">/home/hdfs/ zookeeper-3.2.1</span>和<span lang="EN-US">/home/hdfs/ zookeeper</span>拷贝到其余所有主机的<span lang="EN-US">/home/hdfs</span>目录下。<span lang="EN-US"><br><br></span>(<span lang="EN-US">4</span>)参照<span lang="EN-US">zoo.cfg</span>中的配置,在各主机<span lang="EN-US">myid</span>文件中写入各自的编号。如:<span lang="EN-US">10.192.1.1</span>入<span lang="EN-US">1,10.192.1.2</span>写入<span lang="EN-US">2<br><br></span>(<span lang="EN-US">5</span>)在所有节点上执行<span lang="EN-US">bin/zkServer.sh start</span>,分别启动。<span lang="EN-US"><br><br></span>执行<span lang="EN-US">bin/zkCli.sh -server xxx.xxx.xxx.xxx:2181</span>,检查指定服务器是否成功启动。<span lang="EN-US"><br><br></span>安装和配置<span lang="EN-US">HBase<br><br></span>下载<span lang="EN-US">HBase0.20.1</span>版本,解压到<span lang="EN-US">namenode</span>节点的<span lang="EN-US">/home/hdfs</span>目录下。<span lang="EN-US"><br></span>配置说明<span lang="EN-US"><br><br></span>(<span lang="EN-US">1</span>)系统所有配置项的默认设置在<span lang="EN-US">hbase-default.xml</span>中查看,如果需要修改配置项的值,在<span lang="EN-US">hbase-site.xml</span>中添加配置项。<span lang="EN-US"><br><br></span>在分布式模式下安装<span lang="EN-US">HBase</span>,需要添加的最基本的配置项如下:<span lang="EN-US"><br><br>&lt;property&gt;<br><br>&lt;name&gt;hbase.rootdir&lt;/name&gt;<br><br>&lt;value&gt;hdfs://namenode.hdfs:54310/hbase&lt;/value&gt;<br><br>&lt;description&gt;The directory shared by region servers.&lt;/description&gt;<br><br>&lt;/property&gt;<br><br>&lt;property&gt;<br><br>&lt;name&gt;hbase.cluster.distributed&lt;/name&gt;<br><br>&lt;value&gt;true&lt;/value&gt;<br><br>&lt;description&gt;The mode the cluster will be in. Possible values are<br><br>false: standalone and pseudo-distributed setups with managed Zookeeper<br><br>true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)<br><br>&lt;/description&gt;<br><br>&lt;/property&gt;<br><br></span>(<span lang="EN-US">2</span>)在<span lang="EN-US">conf/hbase-env.sh</span>中修改添加配置项:<span lang="EN-US"><br><br>export JAVA_HOME=/usr/java/jdk1.6.0_16<br><br>export HBASE_MANAGES_ZK=false<br><br>export HBASE_CLASSPATH=/home/hdfs/hadoop-0.20.1/conf<br><br></span>并把<span lang="EN-US">~/hadoop-0.20.1/conf/hdfs-site.xml</span>拷贝至<span lang="EN-US">~/hbase-3.2.1/conf/</span>目录下。<span lang="EN-US"><br><br></span>(<span lang="EN-US">3</span>)将<span lang="EN-US">ZooKeeper</span>的配置文件<span lang="EN-US">zoo.cfg</span>添加到<span lang="EN-US">HBase</span>所有主机的<span lang="EN-US">CLASSPATH</span>中。<span lang="EN-US"><br><br></span>(<span lang="EN-US">4</span>)在<span lang="EN-US">conf/regionservers</span>中添加<span lang="EN-US">hadoop-0.20.1/conf/slaves</span>中所有的<span lang="EN-US">datanode</span>节点。<span lang="EN-US"><br></span>启动<span lang="EN-US"><br><br>Hadoop</span>、<span lang="EN-US">ZooKeeper</span>和<span lang="EN-US">HBase</span>之间应该按照顺序启动和关闭:启动<span lang="EN-US">Hadoop—&gt;</span>启动<span lang="EN-US">ZooKeeper</span>集群<span lang="EN-US">—&gt;</span>启动<span lang="EN-US">HBase—&gt;</span>停止<span lang="EN-US">HBase—&gt;</span>停止<span lang="EN-US">ZooKeeper</span>集群<span lang="EN-US">—&gt;</span>停止<span lang="EN-US">Hadoop</span>。<span lang="EN-US"><br><br></span>在<span lang="EN-US">namenode</span>节点执行<span lang="EN-US">bin/hbase-daemon.sh</span>,启动<span lang="EN-US">master</span>。执行<span lang="EN-US">bin/start-hbase.sh</span>和<span lang="EN-US">bin/stop-hbase.sh </span>脚本启动和停止<span lang="EN-US">HBase</span>服务。<span lang="EN-US"><br></span>接口说明<span lang="EN-US"><br><br>HBase</span>按列存储结构化数据,支持建表、插入记录、查询记录、删除记录和索引操作等等,不支持连接和更新操作。<span lang="EN-US"><br></span>开发步骤<span lang="EN-US"><br><br></span>引入<span lang="EN-US">JAR</span>包<span lang="EN-US"><br><br></span>在<span lang="EN-US">Windows</span>客户端编写<span lang="EN-US">JAVA</span>程序操作<span lang="EN-US">HBase</span>,需要引入一些<span lang="EN-US">JAR</span>包。需要引入的<span lang="EN-US">JAR</span>如下:<span lang="EN-US">hadoop-0.20.1-core.jar</span>,<span lang="EN-US">commons-logging-1.0.4.jar</span>,<span lang="EN-US">commons-logging-api-1.0.4.jar</span>,<span lang="EN-US">zookeeper-3.2.1.jar</span>,<span lang="EN-US">hbase-0.20.1.jar</span>,<span lang="EN-US">log4j-1.2.15.jar</span>。<span lang="EN-US"><br><br></span>开发模式<span lang="EN-US"><br><br></span>在分布式模式下开发,在程序中配置与<span lang="EN-US">HDFS</span>和<span lang="EN-US">ZooKeeper</span>的连接,即可对数据进行操作。<span lang="EN-US"><br>view plaincopy to clipboardprint?<br>import java.util.Date;<br>import java.text.SimpleDateFormat;<br>import java.io.IOException;<br>import java.awt.List;<br>import java.util.Map;<br>import java.util.NavigableMap;<br><br>import org.apache.hadoop.hbase.HBaseConfiguration;<br>import org.apache.hadoop.hbase.HTableDescriptor;<br>import org.apache.hadoop.hbase.HColumnDescriptor;<br>import org.apache.hadoop.hbase.client.HBaseAdmin;<br>import org.apache.hadoop.hbase.client.HTable;<br><br>import org.apache.hadoop.hbase.client.Put;<br>import org.apache.hadoop.hbase.client.Get;<br>import org.apache.hadoop.hbase.client.Scan;<br>import org.apache.hadoop.hbase.client.ResultScanner;<br>import org.apache.hadoop.hbase.client.Delete;<br>import org.apache.hadoop.hbase.client.Result;<br>import org.apache.hadoop.hbase.util.Bytes;<br><br>public class HBaseManager {<br>public static void main(String[] args) throws Exception{<br>HBaseManager manager = new HBaseManager();<br>manager.testGet();<br>}<br><br>public void testQueryRS()throws Exception{<br>HBaseConfiguration config = new HBaseConfiguration();<br>config.set("hbase.master", "10.192.1.1:60000");<br>config.set("hbase.zookeeper.quorum", "10.192.1.1");<br><br>HTable table = new HTable(config, "commodity");<br><br>System.out.println("Get Spin's commodity info");<br><br>Scan scanner = new Scan();<br>scanner.addColumn(Bytes.toBytes("description"));<br>scanner.setMaxVersions();&nbsp; <br>ResultScanner rsScanner = table.getScanner(scanner);<br>System.out.println(rsScanner.toString());<br>Result rs = rsScanner.next();<br>while(null != rs){<br>System.out.println(rs.size());<br>NavigableMap&lt;byte[],NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt;&gt; nMap = rs.getMap();&nbsp; <br>NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt; columnMap = nMap.get(Bytes.toBytes("description"));&nbsp; <br>NavigableMap&lt;Long,byte[]&gt; qualMap = columnMap.get(new byte[]{});&nbsp; <br><br>if(qualMap.entrySet().size() &gt; 0){<br>System.out.println("---------------------------");<br>for(Map.Entry&lt;Long, byte[]&gt; m :qualMap.entrySet())&nbsp; <br>{&nbsp;&nbsp;&nbsp; <br>System.out.println("Value:"+ new String(m.getValue())); <br>}<br>}<br>rs = rsScanner.next();<br>}<br>}<br><br>public void testQueryCommodity()throws Exception{<br>HBaseConfiguration config = new HBaseConfiguration();<br>config.set("hbase.master", "10.192.1.1:60000");<br>config.set("hbase.zookeeper.quorum", "10.192.1.1.203");<br><br>HTable table = new HTable(config, "commodity");<br><br>System.out.println("Get Spin's commodity info");<br>Get mathGet = new Get(new String("Spin").getBytes());<br>mathGet.addColumn(Bytes.toBytes("widgetname"));<br>mathGet.setMaxVersions();&nbsp; <br>Result rs = table.get(mathGet);&nbsp; <br><br>NavigableMap&lt;byte[],NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt;&gt; nMap = rs.getMap();&nbsp; <br>NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt; columnMap = nMap.get(Bytes.toBytes("widgetname"));&nbsp; <br>NavigableMap&lt;Long,byte[]&gt; qualMap = columnMap.get(new byte[]{});&nbsp; <br><br>if(qualMap.entrySet().size() &gt; 0){<br>for(Map.Entry&lt;Long, byte[]&gt; m :qualMap.entrySet())&nbsp; <br>{&nbsp;&nbsp;&nbsp; <br>System.out.println("Value:"+ new String(m.getValue()));&nbsp; <br><br>break;<br>}<br>}<br>}<br><br>public void test()throws Exception{<br>HBaseConfiguration config = new HBaseConfiguration();<br>config.set("hbase.master", "10.192.1.1:60000");<br>config.set("hbase.zookeeper.quorum", "10.192.1.1");<br>HBaseAdmin admin = new HBaseAdmin(config);<br>HTable table = new HTable(config, "scores");<br><br>if (admin.tableExists("scores")){<br>System.out.println("drop table");<br>admin.disableTable("scores");<br>admin.deleteTable("scores");<br>}<br><br>System.out.println("create table");<br>HTableDescriptor tableDescripter = new HTableDescriptor("scores".getBytes());<br>tableDescripter.addFamily(new HColumnDescriptor("grade"));<br>tableDescripter.addFamily(new HColumnDescriptor("course"));<br><br>admin.createTable(tableDescripter);<br><br>System.out.println("add Tom's data");<br><br>Put tomPut = new Put(new String("Tom").getBytes());<br>tomPut.add(new String("grade").getBytes(), new byte[]{}, new String("1").getBytes());<br>tomPut.add(new String("grade").getBytes(), new String("math").getBytes(), new String("87").getBytes());<br>tomPut.add(new String("course").getBytes(), new String("math").getBytes(), new String("97").getBytes());<br>table.put(tomPut);<br><br>System.out.println("add Jerry's data");<br><br>Put jerryPut = new Put(new String("Jerry").getBytes());<br>jerryPut.add(new String("grade").getBytes(), new byte[]{}, new String("2").getBytes());<br>jerryPut.add(new String("grade").getBytes(), new String("math").getBytes(), new String("77").getBytes());<br>jerryPut.add(new String("course").getBytes(), new String("math").getBytes(), new String("92").getBytes());<br>table.put(jerryPut);<br><br>System.out.println("Get Tom's data");<br>Get tomGet = new Get(new String("Tom").getBytes());<br>Result tomResult = table.get(tomGet);<br>System.out.println(tomResult.toString());<br><br>System.out.println("Get Tom's Math grade");<br>Get mathGet = new Get(new String("Tom").getBytes());<br>mathGet.addColumn(Bytes.toBytes("grade"));<br>mathGet.setMaxVersions();&nbsp; <br>Result rs = table.get(mathGet);&nbsp; <br><br>NavigableMap&lt;byte[],NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt;&gt; nMap = rs.getMap();&nbsp; <br>NavigableMap&lt;byte[],NavigableMap&lt;Long,byte[]&gt;&gt; columnMap = nMap.get(Bytes.toBytes("grade"));&nbsp; <br>NavigableMap&lt;Long,byte[]&gt; qualMap = columnMap.get(Bytes.toBytes("math"));&nbsp; <br><br>for(Map.Entry&lt;Long, byte[]&gt; m :qualMap.entrySet())&nbsp; <br>{&nbsp; <br><br>System.out.println("TimeStamp:"+m.getKey());&nbsp; <br>System.out.println("Value:"+ new String(m.getValue()));&nbsp; <br>}<br><br>System.out.println("Delete a column");<br>Delete deleteArt = new Delete(Bytes.toBytes("Tom"));<br>deleteArt.deleteColumn(Bytes.toBytes("grade"), Bytes.toBytes("math"));<br>table.delete(deleteArt);<br>}<br><br>public void testScanner() throws IOException{<br>HBaseConfiguration config = new HBaseConfiguration();<br>config.set("hbase.master", "10.192.1.1:60000");<br>config.set("hbase.zookeeper.quorum", "10.192.1.1");<br><br>HTable table = new HTable(config, "commodity");<br><br>System.out.println("Scan commodity info");<br><br>Scan scanner = new Scan();<br>scanner.addColumn(Bytes.toBytes("widgetname"));<br>scanner.addColumn(Bytes.toBytes("filename"));<br>scanner.addColumn(Bytes.toBytes("description"));<br>scanner.addColumn(Bytes.toBytes("createtime"));<br>//scanner.setMaxVersions();<br>//scanner.setMaxVersions(4);<br>ResultScanner rsScanner = table.getScanner(scanner);<br><br>Result rs = rsScanner.next();<br>for(;null != rs; rs = rsScanner.next()){<br>System.out.println("rs.getRow()[" + new String(rs.getRow()) + "]");<br>System.out.println("[" + new String(rs.getValue(Bytes.toBytes("widgetname"))) + "]");<br>System.out.println("[" + new String(rs.getValue(Bytes.toBytes("filename"))) + "]");<br>System.out.println("[" + new String(rs.getValue(Bytes.toBytes("description"))) + "]");<br>String timeStr = new String(rs.getValue(Bytes.toBytes("createtime")));<br>System.out.println("[" + timeStr + "]");<br><br>SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");<br>try{<br>Date after = dateFormat.parse(timeStr);<br>System.out.println(after);<br>}<br>catch(Exception exp){<br>exp.printStackTrace();<br>}<br>}<br>}<br><br>public void testGet()throws IOException{<br>HBaseConfiguration config = new HBaseConfiguration();<br>config.set("hbase.master", "10.192.1.1:60000");<br>config.set("hbase.zookeeper.quorum", "10.192.1.1");<br><br>HTable table = new HTable(config, "commodity");<br><br>Get get = new Get(new String("xxxx.wgt").getBytes());<br>get.addColumn(Bytes.toBytes("widgetname"));<br>get.addColumn(Bytes.toBytes("filename"));<br>get.addColumn(Bytes.toBytes("description"));<br>get.addColumn(Bytes.toBytes("createtime"));<br><br>get.setMaxVersions(2);<br>System.out.println("00000000000000");<br>Result dbResult = table.get(get);<br><br>System.out.println("11111111111111");<br>System.out.println(dbResult.size());<br>System.out.println("2222222222222222");<br>System.out.println(new String(dbResult.value()));<br>System.out.println("3333333333333333");<br>System.out.println(dbResult.containsColumn(Bytes.toBytes("description"), new byte[]{}));<br>System.out.println("44444444444444444");<br>System.out.println(dbResult.isEmpty());<br>System.out.println("55555555555555555");<br>System.out.println(dbResult.list());<br>System.out.println("66666666666666666");<br>System.out.println(dbResult.raw());<br>System.out.println("77777777777777777");<br>System.out.println(dbResult.toString());<br>System.out.println("88888888888888888");<br>System.out.println(dbResult.raw().clone());<br>System.out.println("99999999999999999");<br>}<br>}</span></span></p></td></tr></tbody></table></p>
<p style="MARGIN: 0cm 0cm 0pt" class="MsoNormal"><span lang="EN-US"><font face="Times New Roman">&nbsp;</font></span></p>
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP