重大事故！IO問題引發線上20臺機器同時崩潰

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"幾年前的一個下午，公司裏碼農們正在安靜地敲着代碼，突然很多人的手機同時“嗶嗶”地響了起來。本來以爲發工資了，都挺高興！打開一看，原來是告警短信"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"故障回顧"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"告警提示“線程數過多，超出閾值”，“CPU空閒率太低”。打開監控系統一看，訂單服務所有20個服務節點都不行了，服務沒響應。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個springboot節點線程數全都達到了最大值。但是JVM堆內存和GC沒有明顯異常。CPU 空閒率基本都是0%，但是CPU使用率並不高，反而IO等待卻非常高。下面是執行top命令查看CPU狀況的截圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b783c10cf0cf4d92e1193f7110d88afd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上圖，我們可以看到:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU空閒率是0%（上圖中紅框id）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU使用率是22%（上圖中紅框 us 13% 加上 sy 9%，us可以理解成用戶進程佔用的CPU，sy可以理解成系統進程佔用的CPU）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU 在等待磁盤IO操作上花費的時間佔比是76.6% （上圖中紅框 wa）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到現在可以確定，問題肯定發生在IO等待上。利用監控系統和jstack命令，最終定位問題發生在文件寫入上。大量的磁盤讀寫導致了JVM線程資源耗盡（注意，不代表系統CPU耗盡）。最終導致訂單服務無法響應上游服務的請求。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"IO，你不知道的那些事兒"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"既然IO對系統性能和穩定性影響這麼大，我們就來深入探究一下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所謂的I/O（Input/Output）操作實際上就是輸入輸出的數據傳輸行爲。程序員最關注的主要是磁盤IO和網絡IO，因爲這兩個IO操作和應用程序的關係最直接最緊密。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"磁盤IO：磁盤的輸入輸出，比如磁盤和內存之間的數據傳輸。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網絡IO：不同系統間跨網絡的數據傳輸，比如兩個系統間的遠程接口調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面這張圖展示了應用程序中發生IO的具體場景："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/639b87dd29da45789d25cd2275eb47cf.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上圖，我們可以瞭解到IO操作發生的具體場景。一個請求過程可能會發生很多次的IO操作："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"頁面請求到服務器會發生網絡IO"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"服務之間遠程調用會發生網絡IO"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"應用程序訪問數據庫會發生網絡IO"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"數據庫查詢或者寫入數據會發生磁盤IO"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"IO和CPU的關係"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不少攻城獅會這樣理解，如果CPU空閒率是0%，就代表CPU已經在滿負荷工作，沒精力再處理其他任務了。真是這樣的嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先看一下計算機是怎麼管理磁盤IO操作的。計算機發展早期，磁盤和內存的數據傳輸是由CPU控制的，也就是說從磁盤讀取數據到內存中，是需要CPU存儲和轉發的，期間CPU一直會被佔用。我們知道磁盤的讀寫速度遠遠比不上CPU的運轉速度。這樣在傳輸數據時就會佔用大量CPU資源，造成CPU資源嚴重浪費。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後來有人設計了一個IO控制器，專門控制磁盤IO。當發生磁盤和內存間的數據傳輸前，CPU會給IO控制器發送指令，讓IO控制器負責數據傳輸操作，數據傳輸完IO控制器再通知CPU。因此，從磁盤讀取數據到內存的過程就不再需要CPU參與了，CPU可以空出來處理其他事情，大大提高了CPU利用率。這個IO控制器就是“DMA”，即直接內存訪問，Direct Memory Access。現在的計算機基本都採用這種DMA模式進行數據傳輸。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/acca2772cc57791bae565d0938ce1991.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上面內容我們瞭解到，IO數據傳輸時，是不佔用CPU的。當應用進程或線程發生IO等待時，CPU會及時釋放相應的時間片資源並把時間片分配給其他進程或線程使用，從而使CPU資源得到充分利用。所以，假如CPU大部分消耗在IO等待（wa）上時，即便CPU空閒率（id）是0%，也並不意味着CPU資源完全耗盡了，如果有新的任務來了，CPU仍然有精力執行任務。如下圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/71/71ef03a4e478941e26db07f2550857e8.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在DMA模式下執行IO操作是不佔用CPU的，所以CPU IO等待（上圖的wa）實際上屬於CPU空閒率的一部分。所以我們執行top命令時，除了要關注CPU空閒率，CPU使用率（us，sy），還要關注IO Wait（wa）。注意，wa只代表磁盤IO Wait，不包括網絡IO Wait。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Java中線程狀態和IO的關係"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當我們用jstack查看Java線程狀態時，會看到各種線程狀態。當發生IO等待時（比如遠程調用時），線程是什麼狀態呢，Blocked還是Waiting？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"答案是Runnable狀態，是不是有些出乎意料！實際上，在操作系統層面Java的Runnable狀態除了包括Running狀態，還包括Ready（就緒狀態，等待CPU調度）和IO Wait等狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/40ad78bae96b9200d8efe62be84de078.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖，Runnable狀態的註解明確說明了，在JVM層面執行的線程，在操作系統層面可能在等待其他資源。如果等待的資源是CPU，在操作系統層面線程就是等待被CPU調度的Ready狀態；如果等待的資源是磁盤網卡等IO資源，在操作系統層面線程就是等待IO操作完成的IO Wait狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有人可能會問，爲什麼Java線程沒有專門的Running狀態呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前絕大部分主流操作系統都是以時間分片的方式對任務進行輪詢調度，時間片通常很短，大概幾十毫秒，也就是說一個線程每次在cpu上只能執行幾十毫秒，然後就會被CPU調度出來變成Ready狀態，等待再一次被CPU執行，線程在Ready和Running兩個狀態間快速切換。通常情況，JVM線程狀態主要爲了監控使用，是給人看的。當你看到線程狀態是Running的一瞬間，線程狀態早已經切換N次了。所以，再給線程專門加一個Running狀態也就沒什麼意義了。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"深入理解網絡IO模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5種Linux網絡IO模型包括：同步阻塞IO、同步非阻塞IO、多路複用IO、信號驅動IO和異步IO。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"寫在前面"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好地理解網絡IO模型，我們先了解幾個基本概念。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Socket（套接字）"},{"type":"text","text":"：Socket可以理解成，在兩個應用程序進行網絡通信時，分別在兩個應用程序中的通信端點。通信時，一個應用程序將數據寫入Socket，然後通過網卡把數據發送到另外一個應用程序的Socket中。我們平常所說的HTTP和TCP協議的遠程通信，底層都是基於Socket實現的。5種網絡IO模型也都要基於Socket實現網絡通信。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"阻塞與非阻塞"},{"type":"text","text":"：所謂阻塞，就是發出一個請求不能立刻返回響應，要等所有的邏輯全處理完才能返回響應。非阻塞反之，發出一個請求立刻返回應答，不用等處理完所有邏輯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"內核空間與用戶空間"},{"type":"text","text":"：在Linux中，應用程序穩定性遠遠比不上操作系統程序，爲了保證操作系統的穩定性，Linux區分了內核空間和用戶空間。可以這樣理解，內核空間運行操作系統程序和驅動程序，用戶空間運行應用程序。Linux以這種方式隔離了操作系統程序和應用程序，避免了應用程序影響到操作系統自身的穩定性。這也是Linux系統超級穩定的主要原因。所有的系統資源操作都在內核空間進行，比如讀寫磁盤文件，內存分配和回收，網絡接口調用等。所以在一次網絡IO讀取過程中，數據並不是直接從網卡讀取到用戶空間中的應用程序緩衝區，而是先從網卡拷貝到內核空間緩衝區，然後再從內核拷貝到用戶空間中的應用程序緩衝區。對於網絡IO寫入過程，過程則相反，先將數據從用戶空間中的應用程序緩衝區拷貝到內核緩衝區，再從內核緩衝區把數據通過網卡發送出去。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"同步阻塞IO"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先看一下傳統阻塞IO。在Linux中，默認情況下所有socket都是阻塞模式的。當用戶線程調用系統函數read()，內核開始準備數據（從網絡接收數據），內核準備數據完成後，數據從內核拷貝到用戶空間的應用程序緩衝區，數據拷貝完成後，請求才返回。從發起read請求到最終完成內核到應用程序的拷貝，整個過程都是阻塞的。爲了提高性能，可以爲每個連接都分配一個線程。因此，在大量連接的場景下就需要大量的線程，會造成巨大的性能損耗，這也是傳統阻塞IO的最大缺陷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d4/d4be4e854fe906be31417e5eb12c5865.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"同步非阻塞IO"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶線程在發起Read請求後立即返回，不用等待內核準備數據的過程。如果Read請求沒讀取到數據，用戶線程會不斷輪詢發起Read請求，直到數據到達（內核準備好數據）後才停止輪詢。非阻塞IO模型雖然避免了由於線程阻塞問題帶來的大量線程消耗，但是頻繁的重複輪詢大大增加了請求次數，對CPU消耗也比較明顯。這種模型在實際應用中很少使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/4829b37274bb043fd6640e466977409a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"多路複用IO模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多路複用IO模型，建立在多路事件分離函數select，poll，epoll之上。在發起read請求前，先更新select的socket監控列表，然後等待select函數返回（此過程是阻塞的，所以說多路複用IO也是阻塞IO模型）。當某個socket有數據到達時，select函數返回。此時用戶線程才正式發起read請求，讀取並處理數據。這種模式用一個專門的監視線程去檢查多個socket，如果某個socket有數據到達就交給工作線程處理。由於等待Socket數據到達過程非常耗時，所以這種方式解決了阻塞IO模型一個Socket連接就需要一個線程的問題，也不存在非阻塞IO模型忙輪詢帶來的CPU性能損耗的問題。多路複用IO模型的實際應用場景很多，比如大家耳熟能詳的Java NIO，Redis以及Dubbo採用的通信框架Netty都採用了這種模型。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/98/98aaa4a908a048e5d984dcb55fb28dc3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是基於select函數Socket編程的詳細流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f0/f0a7f36af6d1b69bac9a5eaa9740dfe0.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"信號驅動IO模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"信號驅動IO模型，應用進程使用sigaction函數，內核會立即返回，也就是說內核準備數據的階段應用進程是非阻塞的。內核準備好數據後向應用進程發送SIGIO信號，接到信號後數據被複制到應用程序進程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用這種方式，CPU的利用率很高。不過這種模式下，在大量IO操作的情況下可能造成信號隊列溢出導致信號丟失，造成災難性後果。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"異步IO模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"異步IO模型的基本機制是，應用進程告訴內核啓動某個操作，內核操作完成後再通知應用進程。在多路複用IO模型中，socket狀態事件到達，得到通知後，應用進程纔開始自行讀取並處理數據。在異步IO模型中，應用進程得到通知時，內核已經讀取完數據並把數據放到了應用進程的緩衝區中，此時應用進程直接使用數據即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很明顯，異步IO模型性能很高。不過到目前爲止，異步IO和信號驅動IO模型應用並不多見，傳統阻塞IO和多路複用IO模型還是目前應用的主流。Linux2.6版本後才引入異步IO模型，目前很多系統對異步IO模型支持尚不成熟。很多應用場景採用多路複用IO替代異步IO模型。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"如何避免IO問題帶來的系統故障"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於磁盤文件訪問的操作，可以採用線程池方式，並設置線程上線，從而避免整個JVM線程池污染，進而導致線程和CPU資源耗盡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於網絡間遠程調用。爲了避免服務間調用的全鏈路故障，要設置合理的TImeout值，高併發場景下可以採用熔斷機制。在同一JVM內部採用線程隔離機制，把線程分爲若干組，不同的線程組分別服務於不同的類和方法，避免因爲一個小功能點的故障，導致JVM內部所有線程受到影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，完善的運維監控（磁盤IO，網絡IO）和APM（全鏈路性能監控）也非常重要，能及時預警，防患於未然，在故障發生時也能幫助我們快速定位問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"看完三件事❤️"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你覺得這篇內容對你還蠻有幫助，我想邀請你幫我三個小忙："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"點贊，轉發，有你們的 『點贊和評論』，纔是我創造的動力。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"關注公衆號 『 "},{"type":"text","marks":[{"type":"strong"}],"text":"java爛豬皮"},{"type":"text","text":" 』，不定期分享原創知識。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"同時可以期待後續文章ing🚀"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/34/34172ad7f3cc8e0f28bd1fc6ca2d2b68.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文作者：馮濤 來自公衆號：架構師進階之路"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}