上次我上網查資料,突然彈出來一個網頁,很黃很暴力,和張殊凡小朋友一樣,我趕緊給關了.不過,從此……我天天上網查資料.今天我就給大家介紹一下我的成果.
比如有一個網站叫做bugzilla.kernel.org,這是一個Linux hacker雲集的網站.
這個網站用於彙報Linux內核的那些bug,每當看到這些bug,Linux黑客們就採用各種手段去解決它,包括暴力手段.
在2007年的那個楓葉飄零的晚秋,一個瑞典人描述了他所遇到的一個bug.這個bug編號爲9335,關於它的更多細節可以在下面這個link中看到.
http://bugzilla.kernel.org/show_bug.cgi?id=9335
當時他是這樣描述的:
Most recent kernel where this bug did not occur: 2.6.23
Distribution: Debian
Hardware Environment: Thinkpad R60, Intel Core 2 Duo
Software Environment: x86_64 kernel, XFS, X.org, KDE.
Problem Description:
When using the system for some time, usually at most a few hours, it suddenly hangs completely, the screen goes black, and it can only be reset with the power switch. The fan is still spinning however and the system seems to generate heat as if it were doing something CPU-intensive.
This happens consistently but at seemingly random times. It's a desktop system, used for some browsing and e-mail mostly.
他說他的系統用着用着就會掛起.後來經高人指點,他開啓了NMI watchdog.於是這次能夠在發生異常的時候打印出oops信息來.下面我們就來從這個oops信息找出問題的根源.
netconsole: network logging started
NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in: netconsole configfs i915 drm rfcomm l2cap xfrm_user xfrm4_tunnel af_key xfrm4_mode_tunnel nfsd exportfs autofs4 cpufreq_conservative cpufreq_userspace cpufreq_stats cpufreq_powersave rpcsec_gss_krb5 auth_rpcgss tunnel4 ipcomp nfs esp4 lockd ah4 nfs_acl sunrpc deflate zlib_deflate twofish_x86_64 twofish_common camellia serpent blowfish des_generic xcbc sha1_generic crypto_null hmac crypto_hash ppp_async crc_ccitt fuse ipv6 ppp_generic nls_utf8 slhc ntfs xfs pl2303 option usbserial kqemu coretemp cpufreq_ondemand acpi_cpufreq freq_table snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_seq_device arc4 ecb ohci_hcd snd_hda_intel snd_pcm_oss snd_mixer_oss iwl3945 snd_pcm pcmcia snd_timer firmware_class snd thinkpad_acpi mousedev mac80211 hci_usb soundcore hwmon serio_raw snd_page_alloc bluetooth video backlight output yenta_socket rsrc_nonstatic pcmcia_core button i2c_i801 cfg80211 pcspkr iTCO_wdt nvram rtc psmouse evdev ext3 jbd mbcache sha256_generic aes_x86_64 aes_generic cbc blkcipher dm_crypt dm_mirror dm_snapshot dm_mod firewire_ohci firewire_core crc_itu_t uhci_hcd ehci_hcd usbcore tg3 sr_mod cdrom sd_mod thermal processor fan
Pid: 0, comm: swapper Not tainted 2.6.24-rc2-melech #3
RIP: 0010:[<ffffffff804876c9>] [<ffffffff804876c9>] _spin_lock+0x59/0x70
RSP: 0018:ffffffff805ecc18 EFLAGS: 00000002
RAX: 0000000000000001 RBX: ffffffff8807afd0 RCX: ffff81000113c8b8
RDX: 0000000000020004 RSI: ffff810004af0028 RDI: 0000000000000001
RBP: ffff81000427cd48 R08: ffff810005a713e0 R09: 00000000ffffff8d
R10: 0000000000000000 R11: ffff810003add80c R12: ffff8100042e2780
R13: ffff810004af0028 R14: ffff81000427cc00 R15: ffff8100042e27b0
FS: 0000000000000000(0000) GS:ffffffff80591000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002ae0240a2a08 CR3: 000000000b81a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff805a2000, task ffffffff8055b340)
Stack: ffff810004af0028 ffffffff88058d60 ffff81000427cdd0 ffffffff8808cbea
00000000042a9940 00000000030007ff ffff8100042e2780 ffff8100042a9ac0
ffff810004af0028 ffff810005a713e8 ffff81000427cd48 ffffffff8808d15a
Call Trace:
<IRQ> [<ffffffff88058d60>] :usbcore:usb_hcd_unlink_urb_from_ep+0x10/0x40
[<ffffffff8808cbea>] :uhci_hcd:uhci_giveback_urb+0x9a/0x220
[<ffffffff8808d15a>] :uhci_hcd:uhci_scan_schedule+0x29a/0x990
[<ffffffff8808f77e>] :uhci_hcd:uhci_irq+0xbe/0x1a0
[<ffffffff880591ed>] :usbcore:usb_hcd_irq+0x2d/0x60
[<ffffffff802752b4>] handle_IRQ_event+0x34/0x70
[<ffffffff80276add>] handle_fasteoi_irq+0x8d/0x110
[<ffffffff8020f9db>] do_IRQ+0x7b/0x100
[<ffffffff8020c921>] ret_from_intr+0x0/0xa
[<ffffffff8048768e>] _spin_lock+0x1e/0x70
[<ffffffff88058d60>] :usbcore:usb_hcd_unlink_urb_from_ep+0x10/0x40
[<ffffffff882d363b>] :ohci_hcd:finish_urb+0x5b/0xe0
[<ffffffff882d37ae>] :ohci_hcd:takeback_td+0xee/0x110
[<ffffffff882d38ac>] :ohci_hcd:dl_done_list+0xdc/0x170
[<ffffffff882d6452>] :ohci_hcd:ohci_irq+0x1e2/0x370
[<ffffffff880591ed>] :usbcore:usb_hcd_irq+0x2d/0x60
[<ffffffff802752b4>] handle_IRQ_event+0x34/0x70
[<ffffffff80276add>] handle_fasteoi_irq+0x8d/0x110
[<ffffffff8020f9db>] do_IRQ+0x7b/0x100
[<ffffffff8020c921>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff880052c0>] :processor:acpi_processor_idle+0x2e0/0x4c0
[<ffffffff880052bc>] :processor:acpi_processor_idle+0x2dc/0x4c0
[<ffffffff88004fe0>] :processor:acpi_processor_idle+0x0/0x4c0
[<ffffffff8020af50>] default_idle+0x0/0x40
[<ffffffff8020aff7>] cpu_idle+0x67/0xd0
[<ffffffff805aabba>] start_kernel+0x2aa/0x330
[<ffffffff805aa117>] _sinittext+0x117/0x120
Code: 8b 03 85 c0 7e f1 eb a3 e8 ba e4 ff ff eb d4 0f 1f 84 00 00
Kernel panic - not syncing: Aiee, killing interrupt handler!
看到RIP的位置在_spin_lock,一個敏感的男人的第一反應就是死鎖.沒錯,這就是個死鎖問題.
我們看到這其中牽涉到的模塊有usbcore,ohci_hcd,以及uhci_hcd.我們注意到函數調用棧裏面有兩個usb_hcd_unlink_urb_from_ep,這個函數來自drivers/usb/core/hcd.c:
1096 /**
1097 * usb_hcd_unlink_urb_from_ep - remove an URB from its endpoint queue
1098 * @hcd: host controller to which @urb was submitted
1099 * @urb: URB being unlinked
1100 *
1101 * Host controller drivers should call this routine before calling
1102 * usb_hcd_giveback_urb(). The HCD's private spinlock must be held and
1103 * interrupts must be disabled. The actions carried out here are required
1104 * for URB completion.
1105 */
1106 void usb_hcd_unlink_urb_from_ep(struct usb_hcd *hcd, struct urb *urb)
1107 {
1108 /* clear all state linking urb to this dev (and hcd) */
1109 spin_lock(&hcd_urb_list_lock);
1110 list_del_init(&urb->urb_list);
1111 spin_unlock(&hcd_urb_list_lock);
1112 }
這裏的確用到了spin_lock,而這把鎖就是hcd_urb_list_lock.那麼是否說明這個問題就已經歸結爲關於hcd_urb_list_lock的deadlock呢?還有待進一步考證.
此時我們還注意到,函數堆棧裏還有兩個很特別的函數,uhci_irq和ohci_irq. 即使是十三歲的張殊凡小朋友都能從這兩個函數的名字判斷得出,這兩個函數分別是uhci主機控制器的中斷服務函數和ohci主機控制器的中斷服務函數.而仔細看這個堆棧,我們不難得出這樣的結論,首先ohci產生了中斷,ohci_irq被調用,但當它正在執行的時候,uhci的中斷髮生了,於是uhci將cpu搶了過去,於是uhci_irq也被調用.
然而問題在於甭管ohci_irq還是uhci_irq,最後都會調用usb_hcd_unlink_urb_from_ep,而這個函數又會調用spin_lock去獲取hcd_urb_list_lock.巧的是當uhci搶cpu的時候,ohci已經獲得了這把鎖,並且還沒來得及釋放.以前我說信號量像北京戶口,你佔了一個名額我就少了一個機會,而這裏不是信號量,是自旋鎖,如果要把自旋鎖打個比方,那麼我把它比作劉濤,很多人YY劉濤,可是她嫁給了200億身價的王柯,做一個最極端的假設,假設你有100億,而且你也幻想得到劉濤,那麼從法律上來講,除非劉濤就是下一個李湘,即除非王柯把她甩了.
而法律應用到這個案例上來,就是說,除非ohci_irq這邊甩了hcd_urb_list_lock,否則uhci_irq那邊就甭想得到.而自旋鎖更絕的地方在於,如果獲得不了鎖,cpu就不停的自旋,它也不睡眠也不幹別的,就好比高衙內得不到林沖的老婆,亦酷似西門大官人得不到潘金蓮,這時他們彷彿丟了魂魄,完全陷入其中,似乎就不能活了一樣,除非違反法律,除非做掉林沖,除非做掉武大郎.所以,這就是一個徹底的死鎖問題,要破除這個很”黃”的死鎖,只能用很暴力的方法.
這個充滿暴力的方法就是把中斷徹底關掉,把spin_lock()換成spin_lock_irqsave(),同時把spin_unlock()換成spin_unlock_irqrestore()就可以.spin_lock_irqsave()是spin_lock()的加強版,它就是在獲得spinlock之前,先關掉本處理器的中斷.用LDD3中的話說就是:
spin_lock_irqsave disables interrupts (on the local processor only) before taking the spinlock.
這樣就使得當我ohci_irq()調用到了usb_hcd_unlink_urb_from_ep()並且進一步,獲取了hcd_urb_list_lock之後,在我沒有釋放它之前,根本就不會允許本cpu再接受別的中斷,任你uhci如何吶喊,如何憤怒,cpu就是不理你,擺出一幅九陽真經中描述的姿態來:他狂由他狂,清風拂山崗.他橫任他橫,明月照大江.
等到cpu願意響應uhci的irq,已然是在spin_unlock_irqrestore()之後.ohci這邊已經把鎖釋放了,它揮一揮衣袖,不帶走一片雲彩.在這種情況下,uhci要獲得就讓他去獲得,天要下雨,娘要嫁人,隨它去吧.
以上就是對這個bug的分析以及解決方案.以上bug存在於2.6.24-rc2和之前的內核中.一個月之後,Alan Stern大俠提交了一個patch以解決這個問題,不過他的解決方法和以上說的略有不同,他不僅僅是讓usb_hcd_unlink_urb_from_ep()內部關中斷,他在註冊中斷服務函數的時候使用了IRQF_DISABLED這個flag,使得ohci_irq/uhci_irq/ehci_irq這些中斷服務函數函數執行的整個過程中都是關中斷的.這樣做當然會引發一些爭議,有人說它很好很強大,有人說它很黃很暴力,不過它是否對系統性能有比較大的影響目前還很難說,讓我們騎驢看唱本—走着瞧.
本文來自CSDN博客，轉載請標明出處：http://blog.csdn.net/fudan_abc/archive/2008/01/17/2049217.aspx