如何優雅的搞垮服務器，再優雅的救活

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"del"}],"text":"故事"},{"type":"text","text":"事故是這樣的"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新開發的jar包部署在老服務器上，版本是Red Hat Enterprise Linux AS release 4 (Nahant Update 5)，提示需要高版本jdk,高版本jdk提示glibc版本太低得升級，是的，就像套娃。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用編譯源碼的方式將glibc由2.3升級到2.9，升級完ls命令不好使了。 用LD_PRELOAD方法解決了ls命令不好使的問題後還挺有成就感的呢！ "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輕度強迫症的我當然要重啓，然後 "}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"#reboot"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就沒有然後了。。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"操作系統起不來了。各種嘗試，最好的結果是卡死在"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"Starting cups-config-daemon:\nStarting HAL daemon:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再也不往下走了。007的服務器被996的程序員幹進了ICU。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看到了吧，搞垮服務器可以顯得很無辜。刪庫顯得太刻意了，會被人指責性格有問題。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"搶救思路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"像《信條》一樣進行一次逆過程，把glibc相關的靜態庫、動態庫都用原來的低版本覆蓋回來。cp覆蓋和安裝rpm覆蓋一起上。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"必要條件"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能進機房，直接操作服務器，因爲ssh此時已經連不上了。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系統光盤，Linux搶救模式需要光盤引導。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系統的iso鏡像文件，用來獲取rpm 【或者替代方法】"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系統的服務器或者虛擬機，用來下載.a文件 【或者替代方法】"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"準備工作"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"rpm安裝包"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將iso文件解壓，在"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"RHEL4.6-i386-AS-DVD\\RedHat\\RPMS"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目錄下就包括所有需要的rpm包。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要準備的安裝包是下面這些： "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f87fcf4a3cbdb08d257c188fd843fddf.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":".a靜態庫文件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到好用的版本一致的服務器對應目錄下載下面的庫文件 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目錄/lib "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3d/3d7ceb0a41a79b229004ed5beaf7e870.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目錄/usr/lib "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/19f226d54b7a0bc5b59d6b093232c65a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將這些安裝包和靜態庫放入一個U盤中，U盤插到服務器上。 "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"搶救過程"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"進入光盤系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將光盤放入光驅。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開機快速按F2，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c38e78b208748b54bcf5730f91f6454.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過+-號調整開機啓動順序，將CD-ROM調整到最上面 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/eca3d48b52c19d03d35f99244170b817.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，系統重新啓動，進入光盤引導界面 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/43/43aac1a4cc6c9eadd7108940c5604fa3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按F5，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/38f08d90eaad66da937cd5d089f25afa.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輸入 linux rescue "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/39/3964fd481d2531f0d793eb7af9517be8.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，稍等一會，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/58/585daf242f0f3063bfb28977e1d68997.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/76e29d11d4c4d594a09863a6f776a23e.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33bf1ddb1b46bbffb6f874f97b2a069a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a9/a97124550cbd7a5e558f906f3e6d623b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將光標移動到No，按回車，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9e/9ed0f3390639c21a74ed2cf379f77884.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回車，進入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/afa725d3a070fab6240b2c9409b4c2b6.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示原有系統已經掛載到/mnt/sysimage，按回車進入，目前所處的就是光盤搶救模式（rescue mode），環境是光盤系統，原系統所有文件都在光盤系統的子目錄/mnt/sysimage裏。 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2d/2dd429ac951136c0fddb564d9bde226f.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到原有系統的所有目錄結構在/mnt/sysimage下都是可以看見的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"掛載U盤"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先將U盤掛載到光盤系統，"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"mount -t vfat /dev/sdb1 /mnt/usb/"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":">不同環境中U盤的標識符不一定是sdb1，在物理機上可能是sda1， 可以通過"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"fdisk –l 命令看各個目錄容量大小來判定哪個是U盤。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果掛載U盤提示格式錯誤，U盤可能是fat16格式，執行 "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mount -t msdos /dev/sdb1 /mnt/usb/ 試試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此時，U盤裏的文件都在/mnt/usb/目錄下， 原系統所有文件都在/mnt/sysimage下。將usb目錄下的文件拷貝到/mnt/sysimage下面你能記住的任意目錄，本文拷貝到/mnt/sysimage/home下。 "}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"cp /mnt/usb/* /mnt/sysimage/home"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"切換到原系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"執行"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"#chroot /mnt/sysimage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個指令使你由當前光盤系統切換到原系統(就是我們要搶救的那個系統)，執行pwd和ls可以看到，你所處的目錄就是原系統的根目錄，賬號是原系統的root賬號。 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48ec6360f948e79b0441de404e64a376.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一句話，原系統直接進進不去，但是從光盤系統跳，是能跳進去的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"安裝rpm"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"進入/home，"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"rpm -ivh --force rpm包名"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個一個安裝U盤的rpm包。裝失敗的就等把成功的都裝完了回頭重試，和答卷子題不會一個玩法，都是依賴關係導致失敗的。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"rpm最好自己重新命名，改成簡短的名字（glibccomm.rpm這種），一定要去掉“-”。我當時看見顯示器上顯示的名字包括亂碼和問號，靠猜來判斷是哪個包，後悔之前沒重命名。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"替換靜態庫文件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後手動cp指令替換/lib 和 /usr/lib的靜態庫（*.a文件）。"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"cp /home/libpthread.a /lib\ncp /home/*.a /usr/lib"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"修改動態庫軟連接"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"手動修改動態庫的軟連接"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論安裝rpm包時是否自動修改過軟連接，都最好手動修改一遍。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到/lib目錄裏，先"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"rm *2.9* #刪除高版本的庫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"ln -sf libutil-2.3.4.solibutil.so.1 \nln -sf libresolv-2.3.4.solibresolv.so.2 \nln -sf libnss_nis-2.3.4.solibnss_nis.so.2 \nln -sf libnss_nisplus-2.3.4.solibnss_nisplus.so.2 \nln -sf libnss_hesiod-2.3.4.solibnss_hesiod.so.2 \nln -sf libnss_files-2.3.4.so libnss_files.so.2 \nln -sf libnss_dns-2.3.4.so libnss_dns.so.2 \nln -sf libnss_compat-2.3.4.solibnss_compat.so.2 \nln -sf libnsl-2.3.4.solibnsl.so.1 \nln -sf libdl-2.3.4.solibdl.so.2 \nln -sf libcrypt-2.3.4.solibcrypt.so.1 \nln -sf libBrokenLocale-2.3.4.solibBrokenLocale.so.1 \nln -sf libanl-2.3.4.solibanl.so.1 \nln -sf libc-2.3.4.solibc.so.6 \nln -sf librt-2.3.4.solibrt.so.1 \nln -sf libpthread-0.10.so libpthread.so.0 \nln -sf libm-2.3.4.solibm.so.6 "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"跳回光盤系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"執行exit跳回到光盤系統， "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/27/27c6f3ac64a21835ac4c3559716a163b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上圖光標處再一次輸入exit，按回車 ，系統會重新啓動。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"立刻修改BIOS，設置系統從硬盤啓動，原系統可以正常啓動了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"完結撒花"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"搶救成功還挺有成就感的呢！其它操作搞垮服務器，也可以試試，只要那個操作能逆向來一遍，問題都不大。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼不重裝？上面部署的東西是多年前放的，物是人非，沒辦法重頭再來。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼敢升級？親眼看見過別人把RHEL6.6的glibc升級了沒出事。真不知道會出這麼嚴重的問題。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果沒有版本一致的光盤，接近的也可以。我實際用的光盤是RHEL4.6，和原系統差了一個小號。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"rpm和.a文件能拿到就行，不用非按本文方法。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網友提供的替換so的方案不靠譜，必須rpm安裝。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.3升級到2.9不可以，不代表升級到2.4也不可以，版本離的近可能成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個服務器至今還在跑着，那些jar包部署到別的服務器上了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}