如何优雅的搞垮服务器，再优雅的救活

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"del"}],"text":"故事"},{"type":"text","text":"事故是这样的"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新开发的jar包部署在老服务器上，版本是Red Hat Enterprise Linux AS release 4 (Nahant Update 5)，提示需要高版本jdk,高版本jdk提示glibc版本太低得升级，是的，就像套娃。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用编译源码的方式将glibc由2.3升级到2.9，升级完ls命令不好使了。 用LD_PRELOAD方法解决了ls命令不好使的问题后还挺有成就感的呢！ "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"轻度强迫症的我当然要重启，然后 "}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"#reboot"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就没有然后了。。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"操作系统起不来了。各种尝试，最好的结果是卡死在"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"Starting cups-config-daemon:\nStarting HAL daemon:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再也不往下走了。007的服务器被996的程序员干进了ICU。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看到了吧，搞垮服务器可以显得很无辜。删库显得太刻意了，会被人指责性格有问题。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"抢救思路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"像《信条》一样进行一次逆过程，把glibc相关的静态库、动态库都用原来的低版本覆盖回来。cp覆盖和安装rpm覆盖一起上。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"必要条件"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能进机房，直接操作服务器，因为ssh此时已经连不上了。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系统光盘，Linux抢救模式需要光盘引导。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系统的iso镜像文件，用来获取rpm 【或者替代方法】"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有相同版本的Linux系统的服务器或者虚拟机，用来下载.a文件 【或者替代方法】"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"准备工作"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"rpm安装包"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将iso文件解压，在"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"RHEL4.6-i386-AS-DVD\\RedHat\\RPMS"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目录下就包括所有需要的rpm包。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要准备的安装包是下面这些： "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f87fcf4a3cbdb08d257c188fd843fddf.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":".a静态库文件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到好用的版本一致的服务器对应目录下载下面的库文件 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目录/lib "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3d/3d7ceb0a41a79b229004ed5beaf7e870.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目录/usr/lib "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/19f226d54b7a0bc5b59d6b093232c65a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将这些安装包和静态库放入一个U盘中，U盘插到服务器上。 "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"抢救过程"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"进入光盘系统"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将光盘放入光驱。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"开机快速按F2，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c38e78b208748b54bcf5730f91f6454.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过+-号调整开机启动顺序，将CD-ROM调整到最上面 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/eca3d48b52c19d03d35f99244170b817.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，系统重新启动，进入光盘引导界面 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/43/43aac1a4cc6c9eadd7108940c5604fa3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按F5，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/38f08d90eaad66da937cd5d089f25afa.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"输入 linux rescue "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/39/3964fd481d2531f0d793eb7af9517be8.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，稍等一会，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/58/585daf242f0f3063bfb28977e1d68997.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/76e29d11d4c4d594a09863a6f776a23e.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33bf1ddb1b46bbffb6f874f97b2a069a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a9/a97124550cbd7a5e558f906f3e6d623b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将光标移动到No，按回车，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9e/9ed0f3390639c21a74ed2cf379f77884.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按回车，进入 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/afa725d3a070fab6240b2c9409b4c2b6.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提示原有系统已经挂载到/mnt/sysimage，按回车进入，目前所处的就是光盘抢救模式（rescue mode），环境是光盘系统，原系统所有文件都在光盘系统的子目录/mnt/sysimage里。 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2d/2dd429ac951136c0fddb564d9bde226f.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到原有系统的所有目录结构在/mnt/sysimage下都是可以看见的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"挂载U盘"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先将U盘挂载到光盘系统，"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"mount -t vfat /dev/sdb1 /mnt/usb/"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":">不同环境中U盘的标识符不一定是sdb1，在物理机上可能是sda1， 可以通过"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"fdisk –l 命令看各个目录容量大小来判定哪个是U盘。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果挂载U盘提示格式错误，U盘可能是fat16格式，执行 "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mount -t msdos /dev/sdb1 /mnt/usb/ 试试"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此时，U盘里的文件都在/mnt/usb/目录下， 原系统所有文件都在/mnt/sysimage下。将usb目录下的文件拷贝到/mnt/sysimage下面你能记住的任意目录，本文拷贝到/mnt/sysimage/home下。 "}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"cp /mnt/usb/* /mnt/sysimage/home"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"切换到原系统"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"执行"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"#chroot /mnt/sysimage"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个指令使你由当前光盘系统切换到原系统(就是我们要抢救的那个系统)，执行pwd和ls可以看到，你所处的目录就是原系统的根目录，账号是原系统的root账号。 "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48ec6360f948e79b0441de404e64a376.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一句话，原系统直接进进不去，但是从光盘系统跳，是能跳进去的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"安装rpm"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"进入/home，"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"rpm -ivh --force rpm包名"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一个一个安装U盘的rpm包。装失败的就等把成功的都装完了回头重试，和答卷子题不会一个玩法，都是依赖关系导致失败的。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"rpm最好自己重新命名，改成简短的名字（glibccomm.rpm这种），一定要去掉“-”。我当时看见显示器上显示的名字包括乱码和问号，靠猜来判断是哪个包，后悔之前没重命名。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"替换静态库文件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后手动cp指令替换/lib 和 /usr/lib的静态库（*.a文件）。"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"cp /home/libpthread.a /lib\ncp /home/*.a /usr/lib"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"修改动态库软连接"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"手动修改动态库的软连接"}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"无论安装rpm包时是否自动修改过软连接，都最好手动修改一遍。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到/lib目录里，先"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"rm *2.9* #删除高版本的库"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后"}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"ln -sf libutil-2.3.4.solibutil.so.1 \nln -sf libresolv-2.3.4.solibresolv.so.2 \nln -sf libnss_nis-2.3.4.solibnss_nis.so.2 \nln -sf libnss_nisplus-2.3.4.solibnss_nisplus.so.2 \nln -sf libnss_hesiod-2.3.4.solibnss_hesiod.so.2 \nln -sf libnss_files-2.3.4.so libnss_files.so.2 \nln -sf libnss_dns-2.3.4.so libnss_dns.so.2 \nln -sf libnss_compat-2.3.4.solibnss_compat.so.2 \nln -sf libnsl-2.3.4.solibnsl.so.1 \nln -sf libdl-2.3.4.solibdl.so.2 \nln -sf libcrypt-2.3.4.solibcrypt.so.1 \nln -sf libBrokenLocale-2.3.4.solibBrokenLocale.so.1 \nln -sf libanl-2.3.4.solibanl.so.1 \nln -sf libc-2.3.4.solibc.so.6 \nln -sf librt-2.3.4.solibrt.so.1 \nln -sf libpthread-0.10.so libpthread.so.0 \nln -sf libm-2.3.4.solibm.so.6 "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"跳回光盘系统"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"执行exit跳回到光盘系统， "}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/27/27c6f3ac64a21835ac4c3559716a163b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上图光标处再一次输入exit，按回车 ，系统会重新启动。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"立刻修改BIOS，设置系统从硬盘启动，原系统可以正常启动了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"完结撒花"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"抢救成功还挺有成就感的呢！其它操作搞垮服务器，也可以试试，只要那个操作能逆向来一遍，问题都不大。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为什么不重装？上面部署的东西是多年前放的，物是人非，没办法重头再来。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为什么敢升级？亲眼看见过别人把RHEL6.6的glibc升级了没出事。真不知道会出这么严重的问题。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果没有版本一致的光盘，接近的也可以。我实际用的光盘是RHEL4.6，和原系统差了一个小号。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"rpm和.a文件能拿到就行，不用非按本文方法。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"网友提供的替换so的方案不靠谱，必须rpm安装。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.3升级到2.9不可以，不代表升级到2.4也不可以，版本离的近可能成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个服务器至今还在跑着，那些jar包部署到别的服务器上了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}