BPF 和 Go: Linux 中的現代內省形式

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文將向你介紹爲什麼我們需要像BPF這樣的東西,並幫助你瞭解何時及如何使用它,以及它是如何幫助作爲工程師的你改進你正在進行的項目的。我們還將研究它與Go相關的一些詳細信息。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/54\/54f298885f5b4869cd408cdd5ae3befe.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"每個人都有自己最喜歡的魔術。對一個人來說他是托爾金(Tolkien),對另一個人來說是普拉切特(Pratchett),對於第三個人來說,比如我,是馬克斯·弗雷(Max Frei)。今天我要給大家講的是我最喜歡的IT魔術:BPF以及圍繞它的現代基礎設施。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/zh.wikipedia.org\/zh-hans\/BPF","title":"xxx","type":null},"content":[{"type":"text","text":"BPF"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前正處於流行的高峯期。這項技術正在飛速發展,深入到了意想不到的領域,並且越來越容易被普通用戶所接受。現在幾乎每個流行的會議都有關於這個主題的演講,早在8月份,我就應邀在GopherCon Russia上做了該主題相關的演講。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我在那裏有過非常好的體驗,所以我想與儘可能多的人分享一下。本文將向你介紹爲什麼我們需要像BPF這樣的東西,並幫助你瞭解何時及如何使用它,以及它是如何幫助作爲工程師的你改進你正在進行的項目的。我們還將研究它與Go相關的一些詳細信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我真正希望的是,你讀完這篇文章後,就像第一次讀完《哈利波特》的小孩兒那樣,眼睛裏閃着光芒,並且希望你能夠親自去嘗試一下這個新“玩具”。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一些背景知識"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好吧,讓一個34歲、留着大鬍子、眼神灼熱的傢伙來告訴你這個魔術是什麼?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我們生活在2020年。打開推特,你可以看到憤怒的技術人員發來的推文,他們都說今天編寫的軟件質量太差了,需要扔掉,我們需要重新開始。有些人甚至威脅說要徹底離開這個行業,因爲他們無法忍受這些,一切都是如此的糟糕、不方便且緩慢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d0\/d04fd1ba87593cc0b853ad3cd736d24b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"他們可能是對的:如果不閱讀上千條評論,就無法確定原因。但有一點我絕對同意,那就是現代軟件堆棧比以往任何時候都要複雜:我們有BIOS、EFI、操作系統、驅動程序、模塊、庫、網絡交互、數據庫、緩存、編排器(如K8s)、Docker容器,最後還有我們自己帶有運行時和垃圾收集器的軟件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一個真正的專業人士可能會花上幾天的時間才能回答這樣一個問題:當在你的瀏覽器中輸入google.com後會發生什麼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"要理解你的系統發生了什麼是非常複雜的,尤其是在當前情況下,出現了問題,你正在賠錢的時候。正是由於這個問題,纔出現了能夠幫助你瞭解系統內部情況的企業。在大公司裏,有的整個部門都是像夏洛克·福爾摩斯(Sherlock holmes)那樣的偵探,他們知道在哪裏敲敲錘子,知道用什麼擰緊螺栓以節省數百萬美元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我喜歡問人們如何在最短的時間內調試出突發問題。通常,人們最先想到的方法是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"分析日誌"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。但問題在於,唯一可訪問的日誌是開發人員放在他們的系統中的日誌,這是很不靈活的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第二種最流行的方法是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"研究度量指標"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。最流行的三個度量指標處理系統都是用Go編寫的。度量指標非常有用,但是,雖然它們確實可以讓你看到症狀,但它們並不總是能夠幫助你定義出問題的根本原因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第三種方法是所謂的“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"可觀察性"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”:你可以對系統的行爲提出儘可能多的複雜問題,並獲得這些問題的答案。由於問題可能會非常複雜,所以答案可能會需要最廣泛的信息,而在問題被提出之前,我們並不知道這些信息是什麼,這意味着可觀察性絕對需要靈活性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"提供一個“動態”更改日誌級別的機會怎麼樣呢?如果使用調試器,在程序運行時連接到程序,並在不中斷程序運行時執行某些操作,又會怎麼樣呢?瞭解哪些查詢會被髮送到系統中,可視化慢查詢的來源,通過pprof查看內存消耗情況,並獲取其隨時間變化的曲線圖呢?如何測量一個函數的延遲以及延遲對參數的依賴性呢?我將所有這些方法都歸爲“可觀測性”這一總稱。這是一套實用程序、方法、知識和經驗,它們結合在一起,共同爲我們提供了機會,即使不能做到我們想做的任何事,至少在系統工作時,它可以在系統中做很多“現場”工作。它相當於現代IT界的一把瑞士軍刀。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d5\/d50afde5664288212699ada344b3ba2b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"但是我們怎樣才能做到這一點呢?市場上已經有很多現成的工具了:有簡單的、有複雜的、有危險的、也有緩慢的。 但是今天這篇文章是關於BPF的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Linux內核是一個事件驅動系統。實際上,在內核以及整個系統中所發生的一切都可以看作是一組事件。中斷是一個事件;通過網絡接收數據包是一個事件;將處理器的控制權轉移到另一個進程是一個事件;運行函數也是一個事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"是的,所以BPF是Linux內核的一個子系統,它使你有機會編寫一些由內核運行以響應事件的小程序。這些程序既可以幫忙你瞭解系統正在發生什麼,也可以用來控制系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"現在讓我們來深入瞭解一下詳細細節。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"什麼是eBPF?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF的第一個版本於1994年問世。有些人在爲tcpdump實用程序編寫用於查看或“嗅探”網絡數據包的簡單規則時,可能會遇到過它。你可以爲tcpdump設置過濾器,這樣你就不必查看所有的內容,只需查看你感興趣的數據包即可。例如,“只查看TCP協議和80端口”。對於每個傳遞的數據包,都會運行一個函數來確定其是否需要保存有問題的特定數據包。可能會有很多數據包,所以我們的函數必須要很快。實際上,我們的tcpdump過濾器被轉換爲BPF函數。下面是一個例子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3bb4bb9dbee19be46a82e34db8b9edb5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#757575","name":"user"}}],"text":"一個簡單的以BPF程序形式呈現的tcpdump過濾器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最初的BPF代表了一個非常簡單帶有多個寄存器的虛擬機。但是,儘管如此,BPF還是大大加快了網絡數據包的過濾速度。在當時,這是一個很重要的進步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/41\/41f17bc24c3f33603223fe77b8b682b8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在2014年,"},{"type":"link","attrs":{"href":"https:\/\/www.linkedin.com\/in\/alexey1\/","title":null,"type":null},"content":[{"type":"text","text":"Alexei Starovoitov"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":",一個非常著名的內核黑客,擴展了BPF的功能。他增加了寄存器的數量和程序允許的大小,添加了JIT編譯,並創建了一個用於檢查程序是否安全的檢查器。然而,最令人印象深刻的是,新的BPF程序不僅能夠在處理數據包時運行,而且還能夠響應其他內核事件,並能在內核和用戶空間之間來回傳遞信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這些變化爲使用BPF的新方法提供了機會。一些過去需要通過編寫複雜而危險的內核模塊來實現的功能,現在可以相對簡單地通過BPF來實現。爲什麼能這麼好呢?是因爲在編寫模塊時,任何錯誤通常都會導致死機(panic),不是“溫和”Go風格的死機,而是內核死機,一旦發生,我們唯一能做的就是重啓。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"普通的Linux用戶突然擁有了一項新的超能力:能夠查看“引擎蓋下”的情況——這是以前只有核心內核開發人員才能使用的東西,或者根本不會提供給任何人。這個選項可以與爲iOS或Android編寫程序的能力相媲美:在舊手機上,這要麼是不可能的,要麼就是要複雜得多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Alexei Starovoitov新版本的BPF被稱爲eBPF(e代表擴展,extended)。但是現在它已經取代了所有舊的BPF用法,並且已經變得非常流行了,爲了簡單起見,它仍然被稱爲BPF。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF可用於何處?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好了,你可以將BPF程序附加到哪些事件或觸發器上呢,人們又是如何開始使用它們以獲取新的能力的呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前,主要有兩大組觸發器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第一組用於處理網絡數據包和管理網絡流量。它們是XDP、流量控制事件及其他幾個事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以下情況需要用到這些事件:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"創建簡單但非常有效的防火牆。Cloudflare和Facebook等公司使用BPF程序來過濾掉大量的寄生流量,並打擊最大規模的DDoS攻擊。由於處理髮生在數據包生命的最早階段,直接在內核中進行(BPF程序的處理有時甚至可以直接推送到網卡中進行),因此可以通過這種方式處理巨量的流量。這些事情過去都是在專門的網絡硬件上完成的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"創建更智能、更有針對性、但性能更好的防火牆——這些防火牆可以檢查通過的流量是否符合公司的規則、是否存在漏洞模式等。例如,Facebook在內部進行這種審計,而一些項目則對外銷售這類產品。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"創建智能負載均衡器。最突出的例子就是Cilium項目,它最常被用作K8s集羣中的網格網絡。Cilium對流量進行管理、均衡、重定向和分析。所有這些都是在內核運行的小型BPF程序的幫助下完成的,以響應這個或那個與網絡數據包或套接字相關的事件。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這是第一組與網絡問題相關並能夠影響網絡通信行爲的觸發器。 第二組則與更普遍的可觀察性相關;在大多數情況下,這組的程序無法影響任何事件,而只能“觀察”。這纔是我更感興趣的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這組的觸發器有如下幾個:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"perf事件(perf events)——與性能和perf Linux分析器相關的事件:硬件處理器計數器、中斷處理、小\/大內存異常攔截等等。例如,我們可以設置一個處理程序,每當內核需要從swap讀取內存頁時,該處理程序就會運行。例如,想象有這樣一個實用程序,它顯示了當前所有使用swap的程序。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"跟蹤點(tracepoints)——內核源代碼中的靜態(由開發人員定義)位置,通過附加到這些位置,你可以從中提取靜態信息(開發人員先前準備的信息)。在這種情況下,靜態似乎是一件壞事,因爲我說過,日誌的缺點之一就是它們只包含了程序員最初放在那裏的內容。從某種意義上說,這是正確的,但跟蹤點有三個重要的優勢:"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有相當多的跟蹤點散落在內核中最有趣的地方"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當它們不“開啓”時,它們不使用任何資源"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"它們是API的一部分,它們是穩定的,不會改變。這非常重要,因爲我們將提到的其他觸發器缺少穩定的API。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"例如,假設有一個關於顯示的實用程序,內核出於某種原因沒有給它時間執行。 你坐着納悶爲什麼它這麼慢,而pprof卻沒有顯示任何什麼有趣的東西。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"USDT——與跟蹤點相同,但是它適用於用戶空間的程序。也就是說,作爲程序員,你可以將這些位置添加到你的程序中。並且許多大型且知名的程序和編程語言都已經採用了這些跟蹤方法:例如MySQL、或者PHP和Python語言。通常,它們的默認設置爲“關閉”,如果要打開它們,需要使用enable-dtrace參數或類似的參數來重新構建解釋器。是的,我們還可以在Go中註冊這種類跟蹤。你可能已經識別出參數名稱中的單詞"},{"type":"link","attrs":{"href":"https:\/\/en.wikipedia.org\/wiki\/DTrace","title":null,"type":null},"content":[{"type":"text","text":"DTrace"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。關鍵在於,這些類型的靜態跟蹤是由"},{"type":"link","attrs":{"href":"https:\/\/en.wikipedia.org\/wiki\/Solaris_(operating_system)","title":null,"type":null},"content":[{"type":"text","text":"Solaris"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"操作系統中誕生的同名系統所推廣的。例如,想象一下,何時創建新線程、何時啓動GC或與特定語言或系統相關的其他內容,我們都能夠知道是怎樣的一種場景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這是另一種魔法開始的地方:"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Ftrace觸發器爲我們提供了在內核的任何函數開始時運行BPF程序的選項。這是完全動態的。這意味着內核將在你選擇的任何內核函數或者在所有內核函數開始執行之前,開始執行之前調用你的BPF函數。你可以連接到所有內核函數,並在輸出時獲取所有調用的有吸引力的可視化效果。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"kprobes\/uprobes提供的功能與ftrace幾乎相同,但在內核和用戶空間中執行函數時,你可以選擇將其附加到任何位置上。如果在函數的中間,變量上有一個“if”,並且能爲這個變量建立一個值的直方圖,那就不是問題。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"kretprobes\/uretprobes——這裏的一切都類似於前面的觸發器,但是它們可以在內核函數或用戶空間中的函數返回時觸發。這類觸發器便於查看函數的返回內容以及測量執行所需的時間。例如,你可以找出“fork”系統調用返回的PID。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我再重複一遍,所有這些最奇妙之處在於,當我們的BPF程序爲了響應這些觸發器而被調用之後,我們可以很好地“環顧四周”:讀取函數的參數,記錄時間,讀取變量,讀取全局變量,進行堆棧跟蹤,保存一些內容以備後用,將數據發送到用戶空間進行處理,和\/或從用戶空間獲取數據或一些其他控制命令以進行過濾。簡直不可思議!"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我不知道你是怎麼想的,但對我來說,這個新的基礎設施就像是一個我很早之間就想要得到的玩具一樣。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"API:怎麼使用它"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好了,讓我們開看一下BPF程序由什麼組成的,以及如何與它交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/63\/636083ff04253a97027a107136a96aa2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"首先,我們有一個BPF程序,如果它通過驗證,就會被加載到內核中。在那裏,它將被JIT編譯器編譯成機器碼,並在內核模式下運行,這時附加的觸發器將會被激活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF程序可以選擇與第二部分交互,即與用戶空間程序交互。有兩種方式可以做到這一點。我們可以向循環緩衝區寫入,而用戶空間程序可以從中讀取。我們也可以對鍵值映射(也就是所謂BPF映射)進行寫入和讀取,相應地,用戶空間程序也可以做同樣的事情,並且相應地,它們就可以相互傳遞信息了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基本用法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"使用BPF最簡單的方法是用C語言編寫BPF程序,然後用Clang編譯器將相關的代碼編譯成虛擬機的代碼(在任何情況下都不應該採用這種從頭開始的方式)。然後,我們直接使用BPF系統調用加載該代碼,並同樣採用BPF系統調用的方式與我們的BPF程序進行交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第一種可用的簡化方法是使用libbpf庫。它是隨內核源代碼一起提供的,允許我們直接使用BPF系統調用。基本上,它爲加載代碼和使用BPF映射將數據從內核發送到用戶空間並返回提供了方便的包裝器。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/iovisor\/bcc","title":null,"type":null},"content":[{"type":"text","text":"bcc"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"顯然,這對人們來說一點也不方便。幸運的是,在iovizor這個品牌下,BCC項目出現了,這使我們的生活變得更加輕鬆了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b5\/b557c5545784caf46bec5122f1605d1f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基本上,它爲我們準備了整個構建環境,並允許我們編寫單個的BPF程序,其中С語言部分會被自動構建並加載到內核中,而用戶空間部分則可以用Python來實現,這既簡單又清晰明瞭。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/iovisor\/bpftrace","title":null,"type":null},"content":[{"type":"text","text":"bpftrace"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"然而,BCC似乎在很多方面都很複雜。出於某些原因,人們特別不喜歡用С語言來寫內核的這部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"同樣那些來自iovizor的人也提供了一個工具,bpftrace,它允許我們用類似於AWK這樣的簡單腳本語言(甚至是單行代碼)來編寫BPF腳本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fb\/fbac146c7db1807a05ef8aa16c1c7963.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Brendan Gregg是生產力和可觀察性領域的知名專家,他對BPF的可用工作方式進行了可視化,如下圖所示:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/02\/02b545daec8f85c209a839a7759abf35.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"縱軸顯示的是給定工具的易用性,而橫軸顯示則是其功能。我們可以看到,BCC是一個非常強大的工具,但它並不是一個超級簡單的工具。而bpftrace要簡單得多,但同時,它的功能則稍弱一些。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"使用BPF的示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"現在,讓我們來看一些具體的例子,看看這些我們可以利用的神奇的力量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BCC和bpftrace都包含了一個“工具”目錄,其中包含了大量的有趣且有用的現成腳本。它們也可以用作本地的Stack Overflow,你可以從中複製代碼塊以用於自己的腳本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"例如,下面是一個顯示DNS查詢延遲的腳本:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─[email protected] ~\n╰─$ sudo gethostlatency-bpfcc\nTIME PID COMM LATms HOST\n16:27:32 21417 DNS Res~ver #93 3.97 live.github.com\n16:27:33 22055 cupsd 7.28 NPI86DDEE.local\n16:27:33 15580 DNS Res~ver #87 0.40 github.githubassets.com\n16:27:33 15777 DNS Res~ver #89 0.54 github.githubassets.com\n16:27:33 21417 DNS Res~ver #93 0.35 live.github.com\n16:27:42 15580 DNS Res~ver #87 5.61 ac.duckduckgo.com\n16:27:42 15777 DNS Res~ver #89 3.81 www.facebook.com\n16:27:42 15777 DNS Res~ver #89 3.76 tech.badoo.com :-)\n16:27:43 21417 DNS Res~ver #93 3.89 static.xx.fbcdn.net\n16:27:43 15580 DNS Res~ver #87 3.76 scontent-frt3-2.xx.fbcdn.net\n16:27:43 15777 DNS Res~ver #89 3.50 scontent-frx5-1.xx.fbcdn.net\n16:27:43 21417 DNS Res~ver #93 4.98 scontent-frt3-1.xx.fbcdn.net\n16:27:44 15580 DNS Res~ver #87 5.53 edge-chat.facebook.com\n16:27:44 15777 DNS Res~ver #89 0.24 edge-chat.facebook.com\n16:27:44 22099 cupsd 7.28 NPI86DDEE.local\n16:27:45 15580 DNS Res~ver #87 3.85 safebrowsing.googleapis.com\n^C%"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這是一個實時顯示DNS查詢完成時間的實用程序,因此,你可以捕獲一些意外的異常值(舉個例子)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"如下則是一個“監視”別人在終端上輸入的內容的腳本:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─[email protected] ~\n╰─$ sudo bashreadline-bpfcc \nTIME PID COMMAND\n16:51:42 24309 uname -a\n16:52:03 24309 rm -rf src\/badoo"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這種腳本可以用來捕獲“壞鄰居”,或者對公司的服務器執行安全審計。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用於查看高級語言調用流的腳本如下所示:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─[email protected] ~\/tmp\n╰─$ sudo \/usr\/sbin\/lib\/uflow -l python 20590\nTracing method calls in python process 20590... Ctrl-C to quit.\nCPU PID TID TIME(us) METHOD\n5 20590 20590 0.173 -> helloworld.py.hello \n5 20590 20590 0.173 -> helloworld.py.world \n5 20590 20590 0.173 helloworld.py.hello \n5 20590 20590 1.174 -> helloworld.py.world \n5 20590 20590 1.174 helloworld.py.hello \n5 20590 20590 2.176 -> helloworld.py.world \n5 20590 20590 2.176 helloworld.py.hello \n6 20590 20590 3.176 -> helloworld.py.world \n6 20590 20590 3.176 helloworld.py.hello \n6 20590 20590 4.177 -> helloworld.py.world \n6 20590 20590 4.177
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章