點個外賣，我把「軟中斷」搞懂了

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近，某團外賣被爆出大數據殺熟，所謂的大數據殺熟指的是平臺利用戶的數據，分析你是否是錢多的人，或者是否是不糾結價格的人，如果是，那麼你買同樣的物品會比普通用戶貴一點，一般這種沒有特地去對比價格是很難發現的，所以平臺就利用了這點額外賺一些錢。說來很可笑，我們作爲平臺的資深用戶，竟然被平臺背後偷偷撈一筆。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不過，大數據殺熟早已是屢見不鮮的事情了，事實上，幾乎所有大平臺都存在這種現象，沒辦法，這就是真實的互聯網。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"剎車，大數據殺熟的話題就說到這了，我們還是迴歸到今日的技術主題：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"什麼是軟中斷？","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bb/bbbef01a2a65fcf16713787c1c33e543.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"中斷是什麼？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先來看看什麼是中斷？在計算機中，中斷是系統用來響應硬件設備請求的一種機制，操作系統收到硬件的中斷請求，會打斷正在執行的進程，然後調用內核中的中斷處理程序來響應請求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣的解釋可能過於學術了，容易雲裏霧裏，我就舉個生活中取外賣的例子。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"小林中午搬完磚，肚子餓了，點了份白切雞外賣，這次我帶閃了，沒有被某團大數據大熟。雖然平臺上會顯示配送進度，但是我也不能一直傻傻地盯着呀，時間很寶貴，當然得去幹別的事情，等外賣到了配送員會通過「電話」通知我，電話響了，我就會停下手中地事情，去拿外賣。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏的打電話，其實就是對應計算機裏的中斷，沒接到電話的時候，我可以做其他的事情，只有接到了電話，也就是發生中斷，我纔會停下當前的事情，去進行另一個事情，也就是拿外賣。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從這個例子，我們可以知道，中斷是一種異步的事件處理機制，可以提高系統的併發處理能力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"操作系統收到了中斷請求，會打斷其他進程的運行，所以","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"中斷請求的響應程序，也就是中斷處理程序，要儘可能快的執行完，這樣可以減少對正常進程運行調度地影響。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且，中斷處理程序在響應中斷時，可能還會「臨時關閉中斷」，這意味着，如果當前中斷處理程序沒有執行完之前，系統中其他的中斷請求都無法被響應，也就說中斷有可能會丟失，所以中斷處理程序要短且快。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還是回到外賣的例子，小林到了晚上又點起了外賣，這次爲了犒勞自己，共點了兩份外賣，一份小龍蝦和一份奶茶，並且是由不同地配送員來配送，那麼問題來了，當第一份外賣送到時，配送員給我打了長長的電話，說了一些雜七雜八的事情，比如給個好評等等，但如果這時另一位配送員也想給我打電話。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很明顯，這時第二位配送員因爲我在通話中（相當於關閉了中斷響應），自然就無法打通我的電話，他可能嘗試了幾次後就走掉了（相當於丟失了一次中斷）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"什麼是軟中斷？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面我們也提到了，中斷請求的處理程序應該要短且快，這樣才能減少對正常進程運行調度地影響，而且中斷處理程序可能會暫時關閉中斷，這時如果中斷處理程序執行時間過長，可能在還未執行完中斷處理程序前，會丟失當前其他設備的中斷請求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那 Linux 系統","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"爲了解決中斷處理程序執行過長和中斷丟失的問題，將中斷過程分成了兩個階段，分別是「上半部和下半部分」","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"上半部用來快速處理中斷","attrs":{}},{"type":"text","text":"，一般會暫時關閉中斷請求，主要負責處理跟硬件緊密相關或者時間敏感的事情。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"下半部用來延遲處理上半部未完成的工作","attrs":{}},{"type":"text","text":"，一般以「內核線程」的方式運行。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面的外賣例子，由於第一個配送員長時間跟我通話，則導致第二位配送員無法撥通我的電話，其實當我接到第一位配送員的電話，可以告訴配送員說我現在下樓，剩下的事情，等我們見面再說（上半部），然後就可以掛斷電話，到樓下後，在拿外賣，以及跟配送員說其他的事情（下半部）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣，第一位配送員就不會佔用我手機太多時間，當第二位配送員正好過來時，會有很大機率撥通我的電話。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再舉一個計算機中的例子，常見的網卡接收網絡包的例子。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網卡收到網絡包後，會通過","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"硬件中斷","attrs":{}},{"type":"text","text":"通知內核有新的數據到了，於是內核就會調用對應的中斷處理程序來響應該事件，這個事件的處理也是會分成上半部和下半部。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上部分要做到快速處理，所以只要把網卡的數據讀到內存中，然後更新一下硬件寄存器的狀態，比如把狀態更新爲表示數據已經讀到內存中的狀態值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接着，內核會觸發一個","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"軟中斷","attrs":{}},{"type":"text","text":"，把一些處理比較耗時且複雜的事情，交給「軟中斷處理程序」去做，也就是中斷的下半部，其主要是需要從內存中找到網絡數據，再按照網絡協議棧，對網絡數據進行逐層解析和處理，最後把數據送給應用程序。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，中斷處理程序的上部分和下半部可以理解爲：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"上半部直接處理硬件請求，也就是硬中斷","attrs":{}},{"type":"text","text":"，主要是負責耗時短的工作，特點是快速執行；","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"下半部是由內核觸發，也就說軟中斷","attrs":{}},{"type":"text","text":"，主要是負責上半部未完成的工作，通常都是耗時比較長的事情，特點是延遲執行；","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有一個區別，硬中斷（上半部）是會打斷 CPU 正在執行的任務，然後立即執行中斷處理程序，而軟中斷（下半部）是以內核線程的方式執行，並且每一個 CPU 都對應一個軟中斷內核線程，名字通常爲「ksoftirqd/CPU 編號」，比如 0 號 CPU 對應的軟中斷內核線程的名字是 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ksoftirqd/0","attrs":{}}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不過，軟中斷不只是包括硬件設備中斷處理程序的下半部，一些內核自定義事件也屬於軟中斷，比如內核調度等、RCU 鎖（內核裏常用的一種鎖）等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"系統裏有哪些軟中斷？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Linux 系統裏，我們可以通過查看 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"/proc/softirqs","attrs":{}}],"attrs":{}},{"type":"text","text":" 的 內容來知曉「軟中斷」的運行情況，以及 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"/proc/interrupts","attrs":{}}],"attrs":{}},{"type":"text","text":" 的 內容來知曉「硬中斷」的運行情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，就來簡單的解析下 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"/proc/softirqs","attrs":{}}],"attrs":{}},{"type":"text","text":" 文件的內容，在我服務器上查看到的文件內容如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7b741e979b3222a99c6c8769d7084e41.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你可以看到，每一個 CPU 都有自己對應的不同類型軟中斷的","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"累計運行次數","attrs":{}},{"type":"text","text":"，有 3 點需要注意下。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一點，要注意第一列的內容，它是代表着軟中斷的類型，在我的系統裏，軟中斷包括了 10 個類型，分別對應不同的工作類型，比如 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NET_RX","attrs":{}}],"attrs":{}},{"type":"text","text":" 表示網絡接收中斷，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NET_TX","attrs":{}}],"attrs":{}},{"type":"text","text":" 表示網絡發送中斷、","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"TIMER","attrs":{}}],"attrs":{}},{"type":"text","text":" 表示定時中斷、","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RCU","attrs":{}}],"attrs":{}},{"type":"text","text":" 表示 RCU 鎖中斷、","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"SCHED","attrs":{}}],"attrs":{}},{"type":"text","text":" 表示內核調度中斷。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二點，要注意同一種類型的軟中斷在不同 CPU 的分佈情況，正常情況下，同一種中斷在不同 CPU 上的累計次數相差不多，比如我的系統裏，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NET_RX","attrs":{}}],"attrs":{}},{"type":"text","text":" 在 CPU0 、CPU1、CPU2、CPU3 上的中斷次數基本是同一個數量級，相差不多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三點，這些數值是系統運行以來的累計中斷次數，數值的大小沒什麼參考意義，但是系統的","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"中斷次數的變化速率","attrs":{}},{"type":"text","text":"纔是我們要關注的，我們可以使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"watch -d cat /proc/softirqs","attrs":{}}],"attrs":{}},{"type":"text","text":" 命令查看中斷次數的變化速率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面提到過，軟中斷是以內核線程的方式執行的，我們可以用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ps","attrs":{}}],"attrs":{}},{"type":"text","text":" 命令可以查看到，下面這個就是在我的服務器上查到軟中斷內核線程的結果：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4a6cb685afa57b8e607ca44f58234314.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以發現，內核線程的名字外面都有有中括號，這說明 ps 無法獲取它們的命令行參數，所以一般來說，名字在中括號裏到，都可以認爲是內核線程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且，你可以看到有 4 個 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ksoftirqd","attrs":{}}],"attrs":{}},{"type":"text","text":" 內核線程，這是因爲我這臺服務器的 CPU 是 4 核心的，每個 CPU 核心都對應着一個內核線程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"如何定位軟中斷 CPU 使用率過高的問題？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要想知道當前的系統的軟中斷情況，我們可以使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"top","attrs":{}}],"attrs":{}},{"type":"text","text":" 命令查看，下面是一臺服務器上的 top 的數據：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/67/67bafbc8b7a24b332fa81fe40b33ca20.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖中的黃色部分 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"si","attrs":{}}],"attrs":{}},{"type":"text","text":"，就是 CPU 在軟中斷上的使用率，而且可以發現，每個 CPU 使用率都不高，兩個 CPU 的使用率雖然只有 3% 和 4% 左右，但是都是用在軟中斷上了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，也可以看到 CPU 使用率最高的進程也是軟中斷 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ksoftirqd","attrs":{}}],"attrs":{}},{"type":"text","text":"，因此可以認爲此時系統的開銷主要來源於軟中斷。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果要知道是哪種軟中斷類型導致的，我們可以使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"watch -d cat /proc/softirqs","attrs":{}}],"attrs":{}},{"type":"text","text":" 命令查看每個軟中斷類型的中斷次數的變化速率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8d86657c3437a1aca6814dc0b14d0f62.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一般對於網絡 I/O 比較高的 Web 服務器，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NET_RX","attrs":{}}],"attrs":{}},{"type":"text","text":" 網絡接收中斷的變化速率相比其他中斷類型快很多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果發現 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NET_RX","attrs":{}}],"attrs":{}},{"type":"text","text":" 網絡接收中斷次數的變化速率過快，接下里就可以使用 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"sar -n DEV","attrs":{}}],"attrs":{}},{"type":"text","text":" 查看網卡的網絡包接收速率情況，然後分析是哪個網卡有大量的網絡包進來。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2c/2cbc6dd0a13235f308818866a2cdef6c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接着，在通過 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"tcpdump","attrs":{}}],"attrs":{}},{"type":"text","text":" 抓包，分析這些包的來源，如果是非法的地址，可以考慮加防火牆，如果是正常流量，則要考慮硬件升級等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了避免由於中斷處理程序執行時間過長，而影響正常進程的調度，Linux 將中斷處理程序分爲上半部和下半部：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上半部，對應硬中斷，由硬件觸發中斷，用來快速處理中斷；","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下半部，對應軟中斷，由內核觸發中斷，用來異步處理上半部未完成的工作；","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux 中的軟中斷包括網絡收發、定時、調度、RCU 鎖等各種類型，可以通過查看 /proc/softirqs 來觀察軟中斷的累計中斷次數情況，如果要實時查看中斷次數的變化率，可以使用 watch -d cat /proc/softirqs 命令。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每一個 CPU 都有各自的軟中斷內核線程，我們還可以用 ps 命令來查看內核線程，一般名字在中括號裏面到，都認爲是內核線程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果在 top 命令發現，CPU 在軟中斷上的使用率比較高，而且 CPU 使用率最高的進程也是軟中斷 ksoftirqd 的時候，這種一般可以認爲系統的開銷被軟中斷佔據了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時我們就可以分析是哪種軟中斷類型導致的，一般來說都是因爲網絡接收軟中斷導致的，如果是的話，可以用 sar 命令查看是哪個網卡的有大量的網絡包接收，再用 tcpdump 抓網絡包，做進一步分析該網絡包的源頭是不是非法地址，如果是就需要考慮防火牆增加規則，如果不是，則考慮硬件升級等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":5}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}