启动hbase失败，执行list命令报错

risepp

今天搭建了hadoop2.6.0+hbase1.1.1+zookeeper-3.4.6，一个namenode加一个datanode，在启动hbase的时候日志里面老是提示如下错误：
2015-09-02 04:04:12,941 FATAL [master:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From master/127.0.0.1 to master:49002 failed on connection exception: java.net.ConnectException: Conne
ction refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

通过hbase shell进入数据库，执行命令list，提示：
hbase(main):001:0> list
TABLE                                                                                                                                 
ERROR: Can't get master address from ZooKeeper; znode data == null

Here is some help for this command:
jps命令输出发现缺少了Hmaster进程
[hadoop@master bin]$ jps
5148 HQuorumPeer
6165 SecondaryNameNode
6947 Jps
6305 ResourceManager
5991 NameNode

hadoop配置文件如下：
core-site.xml
<configuration>  
<property>  
  <name>fs.default.name</name>  
  <value>hdfs://192.168.220.8:9000</value>   
</property>  
<property>  
  <name>dfs.replication</name>   
  <value>1</value>   
</property>  
<property>  
  <name>hadoop.tmp.dir</name>  
  <value>/usr/local/hadoop-2.6.0/tmp</value>   
</property>  
<property>
  <name>io.file.buffer.size</name>
  <value>131702</value>
</property>
</configuration>

mapred-site.xml
<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>192.168.220.8:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>192.168.220.8:19888</value>
</property>
<property>     
        <name>mapred.job.tracker</name>     
        <value>hdfs://192.168.220.8:9001</value>
</property>
</configuration>

yarn-site.xml
<configuration>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
        <name>yarn.resourcemanager.address</name>
        <value>192.168.220.8:8032</value>
</property>
<property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>192.168.220.8:8030</value>
</property>
<property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>192.168.220.8:8031</value>
</property>
<property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>192.168.220.8:8033</value>
</property>
<property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>192.168.220.8:8088</value>
</property>
<property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>768</value>
</property>
</configuration>

hbase配置文件如下：
hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://192.168.220.8:49002/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/usr/local/hadoop-2.6.0/hbase-1.1.1/tmp</value>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/hadoop-2.6.0/zookeeper-3.4.6</value>
</property>
<property>
<name>hbase.zookeeper.property.clientport</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>192.168.220.8</value>
</property>
</configuration>

[root@master hadoop]# hostname
master
[root@master hadoop]# cat /etc/hosts
#127.0.0.1   master master localhost4 localhost4.localdomain4
#::1         master master localhost6 localhost6.localdomain6
192.168.220.8   master
192.168.220.129   s1

[root@s1 ~]# hostname
s1
[root@s1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1              s1.localdomain s1
#::1            localhost6.localdomain6 localhost6
192.168.220.8   master
192.168.220.129   s1

我也重新format过namenode，没用，请高手帮忙看看啥情况，非常感谢

risepp

我重新启动了一下，日志又报了另外一个错误了：
2015-09-02 04:14:26,291 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x14f8dc39d460000, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:22
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:20
        at java.lang.Thread.run(Thread.java:745)
2015-09-02 04:14:26,298 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /192.168.220.8:53737 which had sessionid 0x14f8dc39d460000
2015-09-02 04:15:42,001 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x14f8db0590e0005, timeout of 90000ms exceeded
2015-09-02 04:15:42,002 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x14f8db0590e0004, timeout of 90000ms exceeded
2015-09-02 04:15:42,002 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x14f8db0590e0005


才启动的时候HMaster还在，过了一会儿后就退出了：
[hadoop@master bin]$ jps
7553 HQuorumPeer
6165 SecondaryNameNode
7606 HMaster
7821 Jps
6305 ResourceManager
5991 NameNode
[hadoop@master bin]$ jps
7553 HQuorumPeer
7842 Jps
6165 SecondaryNameNode
6305 ResourceManager
5991 NameNode

risepp

上面的日志信息是zookeeper的日志，不知道是不是zookeeper启动失败导致hbase启动失败。

wenhq

hbase hbck  看下。  接着，重启hbase整个集群。跟zk的关系不大。

risepp

执行该命令后提示的错误信息如下：
2015-09-02 06:46:08,285 WARN  [main] util.HBaseFsck: Encountered exception when opening lock file
java.util.concurrent.ExecutionException: java.net.ConnectException: Call From master/192.168.220.8 to master:49002 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:202)
        at org.apache.hadoop.hbase.util.HBaseFsck.checkAndMarkRunningHbck(HBaseFsck.java:439)
        at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:490)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4665)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4491)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4479)
Caused by: java.net.ConnectException: Call From master/192.168.220.8 to master:49002 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy10.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2753)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2724)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
        at org.apache.hadoop.hbase.util.HBaseFsck$FileLockCallable.call(HBaseFsck.java:379)
        at org.apache.hadoop.hbase.util.HBaseFsck$FileLockCallable.call(HBaseFsck.java:366)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:36
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
        at org.apache.hadoop.ipc.Client.call(Client.java:143
        ... 25 more
2015-09-02 06:46:08,297 ERROR [main] util.HBaseFsck: Another instance of hbck is running, exiting this instance.[If you are sure no other instance is running, delete the lock file null and rerun the tool]
Exception in thread "main" java.io.IOException: Duplicate hbck - Abort
        at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:496)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4665)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4491)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4479)

risepp

另外忘记说了，namenode是在redhat 6.5 64bit上的，datanode是在redhat 5.2 32bit上的，这个环境不一致是否有影响呢？我之前试过在hbase fs -mkdir和hbase fs -put命令创建文件夹和上传文件，都是ok的，从webUI也能看到数据使用空间大小有变化。不过这两个命令成功和hbase貌似没啥关系。

risepp

该问题基本解决了，是由于配置文件填写错误，但是现在遇到一个很神奇的问题，就是在namenode上启动hbase的时候使用start-hbase.sh，regionserver启动失败，日志内容如下：
2015-09-03 00:04:13,589 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:6
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2641)

我将start-hbase.sh启动regionserver的命令打印出来如下：
/usr/local/hadoop-2.6.0/hbase-1.1.1/bin/hbase-daemons.sh --config /usr/local/hadoop-2.6.0/hbase-1.1.1/conf --hosts /usr/local/hadoop-2.6.0/hbase-1.1.1/conf/regionservers start regionserver

但是在regionserver本地执行命令hbase-daemon.sh start regionserver，启动成功，hbase数据库能正常使用并运行list，是啥情况啊？

wenhq

贴配置吧， hbase-site.xml，以及你的regionservers list 以及/etc/hosts
还的明确下是所有的regionserver启动失败？

risepp

hbase-site.xml以及/etc/hosts的内容签名的帖子里面有，region server就只有一台，是自己学习搭建环境，不是生产环境
[root@s1 conf]# cat regionservers
192.168.220.129

今天又试了一下，还是报一样的错误，我把前后相关的error信息都贴出来：
2015-09-05 20:12:21,951 ERROR [regionserver/s1/192.168.220.129:16020] regionserver.HRegionServer: Failed init
java.lang.NoClassDefFoundError: org/htrace/Trace
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:214)
        at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:198
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:111
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1596)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1352)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:89
        at java.lang.Thread.run(Thread.java:745)

2015-09-05 20:12:21,955 FATAL [regionserver/s1/192.168.220.129:16020] regionserver.HRegionServer: ABORTING region server s1,16020,1441509124468: Unhandled: Region server startup failed
java.io.IOException: Region server startup failed
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:2929)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1370)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:89
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:214)
        at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)

2015-09-05 20:12:21,956 FATAL [regionserver/s1/192.168.220.129:16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2015-09-05 20:12:21,977 INFO  [regionserver/s1/192.168.220.129:16020] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "Verbose" : false,
    "NonHeapMemoryUsage" : {
      "committed" : 136445952,
      "init" : 134381568,
      "max" : 167772160,
      "used" : 21850632

2015-09-05 20:12:22,229 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:6
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2641)
2015-09-05 20:12:22,231 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-09-05 20:12:22,234 INFO  [Thread-5] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@140adfb
2015-09-05 20:12:22,234 INFO  [Thread-5] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2015-09-05 20:12:22,238 INFO  [Thread-5] regionserver.ShutdownHook: Shutdown hook finished.

wenhq

Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace????