後起之秀-network policy之eBPF實現

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這篇是Network Policy最後一篇，主題是關於eBPF。前面兩篇，我們聊完了Network Policy的意義和iptables實現，今天我們聊聊如何藉助eBPF來擺脫對iptables的依賴，並實現Network Policy。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章正常是週四更，今年中秋已過去，提前祝大家國慶快樂，來年中秋更快樂！","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"前世","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF的前世是BPF。1992年，Steven McCanne和Van Jacobson寫了一篇論文“The BSD Packet Filter:A New Architecture for User-Level Packet Capture”。在這篇文章裏，作者描述了他們在Unix Kernel裏是如何利用BPF來過濾網絡包的，他們的實現比當時主流的方法快20倍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新方法主要包含了兩個創新：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個工作在內核態的輕量級虛擬機，它可以與CPU寄存器完美契合工作。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲每個application引入了一個專屬的buffer，應用只需要關心與自己相關的package即可。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個令人驚歎的效率提升使得所有的Unix系統都採用了BPF來過濾網絡包，並棄用了傳統的既耗內存效率又低效的方法。BPF至今仍活躍在各類Unix的後繼者身上，包含Linux Kernel。後文將這部分的BPF叫做cBPF（classic BPF）。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"今生","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間來到2014年。Alexei Starovoitov介紹了一種叫extended BPF（eBPF）的設計。新的設計爲匹配最近的硬件做了優化，與cBPF相比，它產生的機器碼執行效率更快，可供使用的寄存器從2個32-bit寄存器大幅提升至10個64-bit的寄存器，這爲基於eBPF來實現更快、更復雜的功能提供了基礎條件。eBPF的速度比cBPF快了4倍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Windows操作系統上著名的Sysinternals套件裏包含了一個系統監控的工具sysmon，它在Linux上的實現也是基於eBPF的。難怪Netflix性能架構師Gregg說BPF是OS內核近50年來最基礎性的改動。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a3/a38cecbd86c181eb0399b3598a5301a5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖 1：eBPF概略圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從這張概略圖中，我們大致可以看出來eBPF項目的一些特點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF program（後文叫eBPF prog）是運行在Kernel裏面的，可以hook到kernel裏面幾乎任何一個函數上，藉助Verifier和JIT的加持，可以安全快速地運行，無需擔心會把系統搞崩潰掉。這點可以完勝kernel module，寫過kernel module的人都記得寫內核驅動時那份如履薄冰的痛苦。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以用它來實現seccomp、觀測、安全控制、網絡流量控制、網路安全、負載均衡、行爲監控等各式各樣的功能。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Map，可以與User space的進程通信。這也就意味着可以通過Map實時、動態地控制eBPF program的行爲，並能及時收集eBPF prog產生的數據。傳統的檢測網絡流量的方法不外乎編寫內核模塊或者從文件系統特定目錄（如/sys/class/net/eth0/statistics/rx_packets）定期讀取數據。每一次讀取意味着一系列文件打開、讀取等費時的系統調用。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux社區提供了各式各樣的toolchain，包括bcc，bpftrace，gobpf，libbpf C/C++ Library，協助你以最小代價方便快捷地編寫eBPF prog。款式各式各樣，總有一個適合你。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面的圖2展示了基於gobpf開發eBPF prog，通過Verifier和JIT後hook到system call的流程。除此之外，圖中還展示了一個eBPF map。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c8e2a36a4cba94f52609e51f5019b08.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖 2：通過SDK gobpf加載eBPF prog、hook system call、map示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是一段簡單的eBPF program代碼。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"c"},"content":[{"type":"text","text":"SEC(\"tracepoint/syscalls/sys_enter_execve\")\nint bpf_prog(void *ctx) {\n char msg[] = \"Hello, BPF World!\";\n bpf_trace_printk(msg, sizeof(msg));\n return 0;\n}\nchar _license[] SEC(\"license\") = \"GPL\";\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過命令","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"clang -O2 -target bpf -c bpf_program.c -o bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":" 即可將其編譯成eBPF prog ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":"是elf格式，.text部分保存的是字節碼，加載到內核且通過Verifier這一關之後，JIT負責將其轉換成機器碼。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過下面的c代碼，可將編譯好的eBPF prog ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":" 加載到內核。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"c"},"content":[{"type":"text","text":"#include \n#include \n#include \"bpf_load.h\"\nint main(int argc, char **argv) {\n if (load_bpf_file(\"bpf_program.o\") != 0) {\n printf(\"The kernel didn't load the BPF program\\n\");\n return -1;\n }\n read_trace_pipe();\n return 0;\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用圖2所示的gobpf的話，就更簡單了。直接調用Go方法","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"func (bpf *Module) AttachTracepoint(name string, fd int) error","attrs":{}}],"attrs":{}},{"type":"text","text":" 加載這段源代碼即可。它會自動完成c代碼轉字節碼的編譯、通過libbpf調用sys_bpf()加載eBPF prog進內核的工作。","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意：這裏是直接使用c源代碼的。傻瓜式的操作方便是方便，但也將一些問題延遲暴露了。比如c代碼如果有編譯問題，只有等調用AttachTracepoint()加載的時候纔會發現。編譯Go代碼的時候，是不會進行c代碼的編譯的。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總體來說，eBPF可以用來做兩大類的事情：tracing和networking。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tracing：顧名思義，這類eBPF prog可以用來幫助你更好地理解你的系統裏發生了什麼。如進程資源使用情況，是否有異常的系統調用行爲等等。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"networking：這類eBPF prog用來檢查和處理系統裏的所有的網絡包。比如可以在網絡包還沒有進入網絡棧的時候就進行導流，繞過iptables進行流量控制，修改IP和端口來實現負載均衡。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體來說，eBPF可以被分爲大概22種的子類別（隨着Kernel的開發，會越來越多）。限於篇幅，這裏就不一一列舉了。詳細內容可參考https://www.man7.org/linux/man-pages/man2/bpf.2.html。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"緣起","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF是個讓人興奮的好東西，而K8s是個讓人亢奮的巨無霸。它們倆的相遇，在Network Policy這個地方擦出了奇妙的火花。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前文我們提到用iptables來實現K8s Network Policy，會使得iptables rule的條目迅速膨脹到上萬條，這會導致網絡包流經網絡棧的時候速度變慢。 如果我們將網絡棧比作河道，網絡包比作水流的話，rule條目的急速增加就像是在河道里插入了一個又一個攔污網，它們在有效過濾網絡包的時候，也顯著降低了流水的速度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過將eBPF替代iptables，能有效改善這種情況。CNI插件Calico和Cilium尤其醉心於此。下面我們以Calico來看看它是如何利用eBPF來替代iptables的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從網絡的角度來看，我們使用eBPF主要是爲了兩個目的：packet capturing 和 filtering。這表示應用程序可以在網絡包流經路徑上插入各種eBPF prog以便來抓取數據包的信息並對特定的網絡包進行各種操作。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Networking data path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在談到eBPF如何替代iptable之前，先讓我們來看下網絡數據路徑的概念。如圖3所示，當網絡設備驅動收到一個網絡包後，XDP會得到最早的機會來接觸這個package，此時它操作的數據結構是xdp_md。XDP全名爲eXpress Data Path。我覺得比較好的翻譯應該是“快速數據路徑”，此處的“快速”作何解釋呢？在圖3中，我特地畫出了一條XDP_TX的路徑，可以看到當滿足特定條件時，它完全避開了tc和協議棧，直接將數據快速地處理掉。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當XDP決定將數據包送往內核做後續處理後，網絡中斷處理程序會申請skb_buff，接下來traffic control(tc)便開始了它的處理流程，也就是我們聽說過的QoS和Queue Descipline。","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意：從這裏開始，tc和內核棧以及其它網絡內核模塊都會以skb_buff爲處理對象。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之後，skb_buff向上流入Networking stack，如果一路暢通，最終會進入應用層。圖中也同樣畫出了當應用層向外發送一個數據的時候，所流經的data path。還記得我們上面的河道比喻嗎？網絡數據包確實如河水一樣，在河道里面流淌。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然這個過程中，iptables依舊位於Networking stack中，我們也沒有必要繞開它，只要不設置過多的iptables rule，便可以快速地穿過iptables這道屏障。","attrs":{}}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"接收數據data path：device driver --> xdp -- >tc(ingress) --> networking stack --> socket --> application","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"發送數據data path：application --> socket --> networking stack --> tc(egress) --> device driver","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e7/e7fccac0d0448c5bbc632766538bec4d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖 3：networking data path關鍵節點示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介紹完網絡數據路徑再來看圖4。別忘了eBPF裏面的字母'F'代表的是Filter。聰明的內核工程師自然是不忘初心，允許我們在網絡數據路徑若干個關鍵節點上hook eBPF來過濾網絡數據。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/83/832456eaab57a83b30bc727ac4dd0cb3.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖4：eBPF 在data path上可以hook的各個關鍵節點示意圖（重點是右側部分，暫時忽略左側）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖4右側部分，從下往上可供hook的eBPF類型至少有如下幾種：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XDP","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"tc","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"socket filter","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kprobe","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tracepoint","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些hook點可以和圖3淺綠色的框中所示的關鍵節點聯繫起來一起看。實際上，可供hook的點還有很多。嗯，老規矩，以後慢慢聊，好吧，我承認，其實是好多我也不會，等我學完一陣子後再來賣。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"calico tc eBPF示例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鋪墊了這麼多，終於到了介紹該如何利用eBPF來實現Network Policy的時候了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是一張利用eBPF hook到tc來實現Network Policy的架構圖。圖中eBPF prog hook在與Pod相連的veth上，它包括3大主要的子program：main prog, policy prog和 epilogue prog。利用eBPF的tail call功能，這3個prog依次被調用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖中eBPF prog會接收到來自物理網卡和節點上其它虛擬設備發過來的traffic。而我們看到policy prog自然地會想到Network Policy。沒錯，通過將Network Policy轉譯成這裏需要的命令，即可方便、快速地控制traffic是否可以流向Pod，而這個過程中我們可以看到iptables被完美地避開了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"強調一下，這裏所說的避開不是說流量不通過iptables（實際上節點上其它虛擬設備發過來的traffic可能不可避免地還是會通過iptables過濾一次），而是說因爲有了tc eBPF的存在，我們便可以不再依賴iptables，不需要創建巨量的iptables rule，從而顯著減低iptables帶來的性能影響。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7a/7a8e16617b1399957dd04c978d619a6e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖5：CNI calico利用eBPF來控制traffic示意圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這張圖裏面的policy prog會引用到一個IP set map。聰明的你一定會想到可以從user space把允許訪問這個Pod的IP和拒絕訪問的IP做成allow list和deny list，然後塞到這個map裏，而policy prog可以根據你的設置來決定是否對traffic放行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"完美的實現！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是本文的全部內容。碼字不易，更多內容請關注二哥的微信公衆號。您的舉手之勞是對二哥莫大的鼓勵。感謝有你！","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/63b236f9f533c70d2e68036f810d0391.jpeg","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}