hbase无法正常启动

risepp

执行start-hbase.sh后，Hmaster进程不存在，检查日志报错如下：
2015-10-14 02:05:32,418 INFO  [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2015-10-14 02:05:32,822 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.HTraceConfiguration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:30
        at java.lang.ClassLoader.loadClass(ClassLoader.java:35
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:574)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:364)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2285)
        ... 5 more

是因为缺少Class么？

risepp

我检查了lib下的文件，是有htrace-core-3.0.4.jar的，且重新从hadoop的lib下拷贝过来后重试，还是报同样的错误

risepp

我查了一下，是少了一个jar包：htrace-core-3.1.0-incubating.jar
拷贝后又报另外的错误了，还是启动失败：
2015-10-14 03:02:12,998 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1313)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1291)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:184)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:177)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:579)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:364)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2285)
        ... 5 more

wenhq

连不上zk，看看zk吧

risepp

今天又试了一下，运行hbase启动命令时直接报错了：
s1: java.io.IOException: Could not find my address: s1 in list of ZooKeeper quorum servers
s1:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.writeMyID(HQuorumPeer.java:149)
s1:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.main(HQuorumPeer.java:70)
starting master, logging to /usr/local/hadoop-2.6.0/hbase-1.1.1/logs/hbase-hadoop-master-master.out
/usr/local/hadoop-2.6.0/hbase-1.1.1/bin/hbase-daemons.sh --config /usr/local/hadoop-2.6.0/hbase-1.1.1/conf --hosts /usr/local/hadoop-2.6.0/hbase-1.1.1/conf/regionservers start regionserver

确实是zookeeper的问题，查看zookeeper的日志，发现有如下故障：
2015-11-04 05:46:35,204 WARN  [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181] quorum.QuorumCnxManager: Cannot open channel to 1 at election
address s1/192.168.220.129:3888
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)

后来去check了一下hbase-site.xml，里面配置的信息没有问题：
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,s1</value>
</property>

直接ping s1也能顺利执行，到底是哪里出了故障啊？之前一直都是ok的

risepp

查了一下，不知道有影响没有啊，hbase-site里面的zookeeper.datadir路径需要和zookeeper的zoo.cfg保持一致，且该路径下的myid需要和zoo.cfg中server.x的内容一致，之前没注意到这两处，修改后重启就好了，真是一个不留神就出问题啊。