公司服務器網絡割接,服務器換了新的IP地址,Hadoop服務器在換完網絡後需要修改IP恢復使用,但是遇到一個問題.
起因是這樣子的,由於服務器前面新加了交換機和路由器,網關掩碼和IP都需要修改,但是我的集羣是三臺Hadoop節點,通過/etc/hosts中添加解析(如: 192.168.1.10 h1)這種,並且修改IP前沒有停止Hadoop服務,就直接斷網開始割接.
網絡割接完成後修改IP重啓了網卡,然後我使用命令
for i in `jps |awk '{print $1}';do kill -9 $i ;done`
殺死了所有進程,確認無誤後開始依次啓動.
我的另一個集羣就是這麼做的,簡直不要太完美的成功了,但是在這裏遇到了問題
在我啓動完zookeeper後啓動Hadoop,打開頁面訪問了 8088端口正常,但是我繼續訪問50070 端口發現大小全部都是0
也就是hdfs沒有啓動成功,然後我就開始看日誌(大部分日誌放在文件結尾),看到這麼一條,然後就停止了.
java.io.IOException: NameNode is not formatted.
他居然提示我格式化namenode?
百度了一下,大部分是小白搭建集羣忘記格式化,執行 format namenode,但是我絕不會是這種情況.
在百度和bing的海洋裏遨遊,我發現所有的解決方式都是格式化namenode 後重啓,但我的是線上環境,裏面存了好多客戶信息,動不動就格式化還怎麼玩
解決方法
懷疑是在停止的時候,或者割接網絡的時候對Hadoop還有寫入數據操作,導致數據結構損壞,通過尋找大佬,大佬早年看Hadoop原理的時候,看到過可能的解決方式.
我的數據是放在 /data 目錄下,這個可以再配置文件中找到
# 我的目錄結構
[root@h1 data]# cd /data/
[root@h1 data]# ls
hadoop hdfs namenode yarn
然後進入Hadoop目錄
[root@h1 data]# cd hadoop/tmp/dfs/namesecondary
[root@h1 dfs]# ls
current current.tar.gz in_use.lock
這裏有個current的目錄,裏面存儲的是備份後的Hadoop namenode節點的數據
示例:
[root@h1 namesecondary]# ls current|head
edits_0000000000000749853-0000000000000749948
edits_0000000000000749949-0000000000000749986
edits_0000000000000749987-0000000000000750115
edits_0000000000000750116-0000000000000750173
edits_0000000000000750174-0000000000000750301
edits_0000000000000750302-0000000000000750333
edits_0000000000000750334-0000000000000750455
edits_0000000000000750456-0000000000000750513
edits_0000000000000750514-0000000000000750590
edits_0000000000000750591-0000000000000750681
不確定數據是否可用,我先將namenode數據備份,直接複製目錄,然後將這裏的所有文件移動到了namenode目錄下覆蓋,再重啓Hadoop
完美啓動成功.
再次百度,感覺這是個類似MySQL的 binlog,又有點類似MySQL主從,Hadoop會先將寫入數據加載到內存,然後修改,修改完成後重新寫入磁盤,並且將操作複製給current一份,作爲備份.具體我也就不太懂了…因爲是後來補的可能會錯過寫細節,大體是這麼個步驟.記錄下這個問題
Hadoop 報錯部分日誌
2020-05-29 09:42:37,362 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: VM type = 64-bit
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: 1.0% max memory 889 MB = 8.9 MB
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: capacity = 2^20 = 1048576 entries
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: ACLs enabled? false
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: XAttrs enabled? true
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Maximum size of an xattr: 16384
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: Computing capacity for map cachedBlocks
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: VM type = 64-bit
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: 0.25% max memory 889 MB = 2.2 MB
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: capacity = 2^18 = 262144 entries
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2020-05-29 09:42:37,624 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2020-05-29 09:42:37,625 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: Computing capacity for map NameNodeRetryCache
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: VM type = 64-bit
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
2020-05-29 09:42:37,627 INFO org.apache.hadoop.util.GSet: capacity = 2^15 = 32768 entries
2020-05-29 09:42:37,636 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data/namenode/in_use.lock acquired by nodename 4270@h1
2020-05-29 09:42:37,638 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:984)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:820)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:804)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
2020-05-29 09:42:37,640 INFO org.mortbay.log: Stopped [email protected]:50070
2020-05-29 09:42:37,741 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2020-05-29 09:42:37,741 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2020-05-29 09:42:37,742 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2020-05-29 09:42:37,742 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:984)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:820)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:804)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
2020-05-29 09:42:37,743 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2020-05-29 09:42:37,744 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h1/20.88.2.114
************************************************************/