hadoop集成kerberos錯誤排查-core dump

最近研究kerberos,剛開始安裝就遇到了個棘手的問題,core dump了。
好些年不寫c程序,這東西還真有點陌生。

[root@hadoop165 ~]# service krb5kdc start
Redirecting to /bin/systemctl start  krb5kdc.service
Job for krb5kdc.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

[root@hadoop165 ~]# systemctl status krb5kdc.service
● krb5kdc.service - Kerberos 5 KDC
   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; enabled; vendor preset: disabled)
   Active: failed (Result: core-dump) since 一 2017-08-28 16:31:20 CST; 4s ago
  Process: 792 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=dumped, signal=SEGV)
 Main PID: 9146 (code=dumped, signal=SEGV)

828 16:31:08 hadoop165 systemd[1]: Starting Kerberos 5 KDC...
828 16:31:20 hadoop165 systemd[1]: krb5kdc.service: control process exited, code=dumped status=11
828 16:31:20 hadoop165 systemd[1]: Failed to start Kerberos 5 KDC.
828 16:31:20 hadoop165 systemd[1]: Unit krb5kdc.service entered failed state.
828 16:31:20 hadoop165 systemd[1]: krb5kdc.service failed.

根據提示又查看了下journalctl -xe。
發現一連串磁盤空間滿的報錯。
df查看還真是滿了。

[root@hadoop165 sbin]# df
文件系統                   1K-塊     已用     可用 已用% 掛載點
/dev/mapper/centos-root 52464612 52464592       20  100% /
devtmpfs                 3995156        0  3995156    0% /dev
tmpfs                    4005608        0  4005608    0% /dev/shm
tmpfs                    4005608   410992  3594616   11% /run
tmpfs                    4005608        0  4005608    0% /sys/fs/cgroup
/dev/mapper/centos-home 94919392   247052 94672340    1% /home
/dev/sda1                 508588   128924   379664   26% /boot
tmpfs                     801124        0   801124    0% /run/user/0
tmpfs                     801124        0   801124    0% /run/user/1000

清理磁盤,空間剩餘42%。
重新測試,還是報原來的錯誤。
systemctl status的信息沒有說明。
繼續查看journalctl -xe的報錯,根據時間點發現了具體的報錯。
這裏發現核心報錯爲db2.so的segment,懷疑是存儲出了問題,結合上面的磁盤空間滿了的情況懷疑數據庫處於一種中間狀態導致程序異常。

[root@hadoop165 logs]# journalctl -xe --no-pager

8月 29 09:45:09 hadoop165 polkitd[931]: Registered Authentication Agent for unix-process:12871:411771162 (system bus name :1.193646 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8)
8月 29 09:45:09 hadoop165 systemd[1]: Starting Kerberos 5 KDC...
-- Subject: Unit krb5kdc.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit krb5kdc.service has begun starting up.
8月 29 09:45:10 hadoop165 kernel: krb5kdc[12917]: segfault at 7fd000053166 ip 00007fd0694cafa6 sp 00007ffd92e568e0 error 6 in db2.so[7fd0694bc000+18000]
8月 29 09:45:58 hadoop165 abrt-hook-ccpp[12919]: /var/spool/abrt is 3445558898 bytes (more than 1279MiB), deleting 'ccpp-2017-08-29-09:41:58-12318'
8月 29 09:45:59 hadoop165 systemd[1]: krb5kdc.service: control process exited, code=dumped status=11
8月 29 09:45:59 hadoop165 systemd[1]: Failed to start Kerberos 5 KDC.
-- Subject: Unit krb5kdc.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit krb5kdc.service has failed.
-- 
-- The result is failed.
8月 29 09:45:59 hadoop165 systemd[1]: Unit krb5kdc.service entered failed state.
8月 29 09:45:59 hadoop165 systemd[1]: krb5kdc.service failed.
8月 29 09:45:59 hadoop165 polkitd[931]: Unregistered Authentication Agent for unix-process:12871:411771162 (system bus name :1.193646, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8) (disconnected from bus)
8月 29 09:46:00 hadoop165 abrt-server[12991]: 正在生成核心轉儲

將數據庫數據清空,並重新生成,測試問題解決。

[root@hadoop165 ~]# cd /var/kerberos/krb5kdc/
[root@hadoop165 krb5kdc]# ls
kadm5.acl  kdc.conf  principal  principal.kadm5  principal.kadm5.lock  principal.ok
[root@hadoop165 krb5kdc]# rm principal* -rf
[root@hadoop165 krb5kdc]# kdb5_util create -r BONC -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'BONC',
master key name 'K/M@BONC'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: 
Re-enter KDC database master key to verify: 
[root@hadoop165 krb5kdc]# 
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章