2020-01-19 16:37:05,531 INFO [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 from any node: java.io.IOException: No live nodes contain block BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
2020-01-19 16:37:05,531 WARN [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 11284.466617084763 msec.
2020-01-19 16:37:16,816 WARN [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2020-01-19 16:37:16,817 WARN [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2020-01-19 16:37:16,817 WARN [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:991)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:638)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:879)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:932)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:524)
at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:515)
at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:491)
at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:174)
at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1428)
at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:49)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
hbase報錯： org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001
最後查看原因是hadoop集羣的幾個datanode節點 宕機
解決：
重啓datanode節點。
start|stop-dfs.sh 在master上使用,否則只會停掉該臺機器的datanode(namenode)
start|stop-yarn.sh(會停掉ResourceManager與NodeManager)在配置ResourceManager的機器上使用,否則只會停止該臺機器上的NodeManager
參考： https://www.cnblogs.com/tele-share/p/9508225.html
