系統環境
CDH 6.3.1
Hue 4.4.0
錯誤現象
Hue 創建hbase 表，遇到如下錯誤：
hue server log:
[06/Jun/2020 13:54:46 +0800] access INFO 192.168.8.69 hue - "POST /hbase/install_examples HTTP/1.1" returned in 120103ms
[06/Jun/2020 13:54:46 +0800] thrift_util INFO Thrift saw exception: HTTPConnectionPool(host='master02-prod.cdh', port=9090): Read timed out. (read timeout=120)
[06/Jun/2020 13:54:46 +0800] thrift_util ERROR Thrift saw exception (this may be expected).
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/thrift_util.py", line 473, in wrapper
ret = res(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/apps/hbase/gen-py/hbased/Hbase.py", line 54, in decorate
return func(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/apps/hbase/gen-py/hbased/Hbase.py", line 957, in createTable
self.send_createTable(tableName, columnFamilies)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/apps/hbase/gen-py/hbased/Hbase.py", line 967, in send_createTable
self._oprot.trans.flush()
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/build/env/lib/python2.7/site-packages/thrift-0.9.1-py2.7-linux-x86_64.egg/thrift/transport/TTransport.py", line 170, in flush
self.__trans.flush()
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/thrift_/http_client.py", line 87, in flush
self._data = self._root.post('', data=data, headers=self._headers)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 157, in post
allow_redirects=allow_redirects, clear_cookies=clear_cookies, log_response=log_response)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 80, in invoke
log_response=log_response)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 99, in _invoke
clear_cookies=clear_cookies)
File "/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 215, in execute
raise self._exc_class(ex)
RestException: HTTPConnectionPool(host='master02-prod.cdh', port=9090): Read timed out. (read timeout=120)
[06/Jun/2020 13:54:42 +0800] access INFO 192.168.8.69 hue - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:42 +0800] resource DEBUG GET /cluster/apps Got response in 2ms: {"apps":{}}
[06/Jun/2020 13:54:41 +0800] access INFO 192.168.8.69 hue - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:41 +0800] resource DEBUG GET /cluster/apps Got response in 2ms: {"apps":{}}
[06/Jun/2020 13:54:41 +0800] access INFO 10.0.101.86 -anon- - "HEAD /desktop/debug/is_alive HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:41 +0800] access DEBUG 10.0.101.86 -anon- - "HEAD /desktop/debug/is_alive HTTP/1.1"
[06/Jun/2020 13:54:39 +0800] access INFO 192.168.8.69 hue - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:39 +0800] resource DEBUG GET /cluster/apps Got response in 2ms: {"apps":{}}
[06/Jun/2020 13:54:32 +0800] thrift_util DEBUG Thrift call: <class 'hbased.Hbase.Client'>.createTable(args=('analytics_demo', [ColumnDescriptor(bloomFilterType='NONE', bloomFilterNbHashes=0, name='hour', maxVersions=3, blockCacheEnabled=False, inMemory=False, timeToLive=-1, bloomFilterVectorSize=0, compression='NONE'), ColumnDescriptor(bloomFilterType='NONE', bloomFilterNbHashes=0, name='day', maxVersions=3, blockCacheEnabled=False, inMemory=False, timeToLive=-1, bloomFilterVectorSize=0, compression='NONE'), ColumnDescriptor(bloomFilterType='NONE', bloomFilterNbHashes=0, name='total', maxVersions=3, blockCacheEnabled=False, inMemory=False, timeToLive=-1, bloomFilterVectorSize=0, compression='NONE')]), kwargs={'doas': u'hue'})
[06/Jun/2020 13:54:20 +0800] access INFO 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/hue-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1" returned in 0ms
[06/Jun/2020 13:54:20 +0800] access DEBUG 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/hue-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1"
[06/Jun/2020 13:54:20 +0800] access INFO 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/hue~notebook-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1" returned in 0ms
[06/Jun/2020 13:54:20 +0800] access DEBUG 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/hue~notebook-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1"
[06/Jun/2020 13:54:20 +0800] access INFO 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/vendors~hue-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:20 +0800] access DEBUG 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/vendors~hue-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1"
[06/Jun/2020 13:54:20 +0800] access INFO 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/vendors~hue~notebook-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1" returned in 3ms
[06/Jun/2020 13:54:20 +0800] access DEBUG 192.168.8.69 hue - "GET /static/desktop/js/bundles/hue/vendors~hue~notebook-bundle-28e313c4f46a25cb539e.js.map HTTP/1.1"
問題定位
hbase master log:
service=MasterService
2020-06-06 14:06:59,269 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:50824, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:07:39,268 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:50926, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:08:04,272 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51018, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:09:09,279 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51228, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:10:14,269 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51396, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:11:14,282 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51568, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:12:19,283 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51736, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:13:11,538 INFO org.apache.hadoop.hbase.mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0
2020-06-06 14:13:12,856 INFO org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=30
2020-06-06 14:13:13,404 ERROR org.apache.hadoop.hbase.master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
2020-06-06 14:13:24,276 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:51908, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:14:24,281 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52080, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:15:29,280 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52248, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:16:34,281 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52434, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:17:34,283 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52610, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:18:34,283 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52786, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:19:39,278 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:52960, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
2020-06-06 14:20:33,185 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.13:43616, version=2.1.0-cdh6.3.1, sasl=false, ugi=root (auth:SIMPLE), service=MasterService
2020-06-06 14:20:39,282 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.0.101.86:53142, version=2.1.0-cdh6.3.1, sasl=false, ugi=hbase (auth:SIMPLE), service=MasterService
cloudera 診斷日誌： /var/log/hbase/hbase-cmf-hbase-HBASETHRIFTSERVER-master02-prod.cdh.log.out：
Sat Jun 06 13:59:57 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 13:59:57 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 13:59:58 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 13:59:59 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 14:00:01 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 14:00:05 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 14:00:15 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 14:00:25 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Sat Jun 06 14:00:35 CST 2020, RpcRetryingCaller{globalStartTime=1591423197351, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave03-prod.cdh,16020,1591413190916
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3305)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1428)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
hase shell 創建table失敗：
參照參考資料3同步服務器時間後問題依舊
參照參考資料4重新初始化hbase，問題解決
參考資料
1、https://blog.csdn.net/mnasd/article/details/80736529
2、https://my.oschina.net/hblt147/blog/1920466
3、https://blog.csdn.net/lieyanhaipo/article/details/77801605
4、https://www.cnblogs.com/yybrhr/p/11378725.html