后起之秀-network policy之eBPF实现

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这篇是Network Policy最后一篇，主题是关于eBPF。前面两篇，我们聊完了Network Policy的意义和iptables实现，今天我们聊聊如何借助eBPF来摆脱对iptables的依赖，并实现Network Policy。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章正常是周四更，今年中秋已过去，提前祝大家国庆快乐，来年中秋更快乐！","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"前世","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF的前世是BPF。1992年，Steven McCanne和Van Jacobson写了一篇论文“The BSD Packet Filter:A New Architecture for User-Level Packet Capture”。在这篇文章里，作者描述了他们在Unix Kernel里是如何利用BPF来过滤网络包的，他们的实现比当时主流的方法快20倍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新方法主要包含了两个创新：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一个工作在内核态的轻量级虚拟机，它可以与CPU寄存器完美契合工作。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为每个application引入了一个专属的buffer，应用只需要关心与自己相关的package即可。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个令人惊叹的效率提升使得所有的Unix系统都采用了BPF来过滤网络包，并弃用了传统的既耗内存效率又低效的方法。BPF至今仍活跃在各类Unix的后继者身上，包含Linux Kernel。后文将这部分的BPF叫做cBPF（classic BPF）。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"今生","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"时间来到2014年。Alexei Starovoitov介绍了一种叫extended BPF（eBPF）的设计。新的设计为匹配最近的硬件做了优化，与cBPF相比，它产生的机器码执行效率更快，可供使用的寄存器从2个32-bit寄存器大幅提升至10个64-bit的寄存器，这为基于eBPF来实现更快、更复杂的功能提供了基础条件。eBPF的速度比cBPF快了4倍。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Windows操作系统上著名的Sysinternals套件里包含了一个系统监控的工具sysmon，它在Linux上的实现也是基于eBPF的。难怪Netflix性能架构师Gregg说BPF是OS内核近50年来最基础性的改动。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a3/a38cecbd86c181eb0399b3598a5301a5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图 1：eBPF概略图","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从这张概略图中，我们大致可以看出来eBPF项目的一些特点：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF program（后文叫eBPF prog）是运行在Kernel里面的，可以hook到kernel里面几乎任何一个函数上，借助Verifier和JIT的加持，可以安全快速地运行，无需担心会把系统搞崩溃掉。这点可以完胜kernel module，写过kernel module的人都记得写内核驱动时那份如履薄冰的痛苦。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以用它来实现seccomp、观测、安全控制、网络流量控制、网路安全、负载均衡、行为监控等各式各样的功能。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过Map，可以与User space的进程通信。这也就意味着可以通过Map实时、动态地控制eBPF program的行为，并能及时收集eBPF prog产生的数据。传统的检测网络流量的方法不外乎编写内核模块或者从文件系统特定目录（如/sys/class/net/eth0/statistics/rx_packets）定期读取数据。每一次读取意味着一系列文件打开、读取等费时的系统调用。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux社区提供了各式各样的toolchain，包括bcc，bpftrace，gobpf，libbpf C/C++ Library，协助你以最小代价方便快捷地编写eBPF prog。款式各式各样，总有一个适合你。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面的图2展示了基于gobpf开发eBPF prog，通过Verifier和JIT后hook到system call的流程。除此之外，图中还展示了一个eBPF map。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c8e2a36a4cba94f52609e51f5019b08.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图 2：通过SDK gobpf加载eBPF prog、hook system call、map示意图","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是一段简单的eBPF program代码。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"c"},"content":[{"type":"text","text":"SEC(\"tracepoint/syscalls/sys_enter_execve\")\nint bpf_prog(void *ctx) {\n char msg[] = \"Hello, BPF World!\";\n bpf_trace_printk(msg, sizeof(msg));\n return 0;\n}\nchar _license[] SEC(\"license\") = \"GPL\";\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过命令","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"clang -O2 -target bpf -c bpf_program.c -o bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":" 即可将其编译成eBPF prog ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":"是elf格式，.text部分保存的是字节码，加载到内核且通过Verifier这一关之后，JIT负责将其转换成机器码。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过下面的c代码，可将编译好的eBPF prog ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"bpf_program.o","attrs":{}}],"attrs":{}},{"type":"text","text":" 加载到内核。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"c"},"content":[{"type":"text","text":"#include \n#include \n#include \"bpf_load.h\"\nint main(int argc, char **argv) {\n if (load_bpf_file(\"bpf_program.o\") != 0) {\n printf(\"The kernel didn't load the BPF program\\n\");\n return -1;\n }\n read_trace_pipe();\n return 0;\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用图2所示的gobpf的话，就更简单了。直接调用Go方法","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"func (bpf *Module) AttachTracepoint(name string, fd int) error","attrs":{}}],"attrs":{}},{"type":"text","text":" 加载这段源代码即可。它会自动完成c代码转字节码的编译、通过libbpf调用sys_bpf()加载eBPF prog进内核的工作。","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意：这里是直接使用c源代码的。傻瓜式的操作方便是方便，但也将一些问题延迟暴露了。比如c代码如果有编译问题，只有等调用AttachTracepoint()加载的时候才会发现。编译Go代码的时候，是不会进行c代码的编译的。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"总体来说，eBPF可以用来做两大类的事情：tracing和networking。","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tracing：顾名思义，这类eBPF prog可以用来帮助你更好地理解你的系统里发生了什么。如进程资源使用情况，是否有异常的系统调用行为等等。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"networking：这类eBPF prog用来检查和处理系统里的所有的网络包。比如可以在网络包还没有进入网络栈的时候就进行导流，绕过iptables进行流量控制，修改IP和端口来实现负载均衡。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具体来说，eBPF可以被分为大概22种的子类别（随着Kernel的开发，会越来越多）。限于篇幅，这里就不一一列举了。详细内容可参考https://www.man7.org/linux/man-pages/man2/bpf.2.html。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"缘起","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"eBPF是个让人兴奋的好东西，而K8s是个让人亢奋的巨无霸。它们俩的相遇，在Network Policy这个地方擦出了奇妙的火花。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前文我们提到用iptables来实现K8s Network Policy，会使得iptables rule的条目迅速膨胀到上万条，这会导致网络包流经网络栈的时候速度变慢。 如果我们将网络栈比作河道，网络包比作水流的话，rule条目的急速增加就像是在河道里插入了一个又一个拦污网，它们在有效过滤网络包的时候，也显著降低了流水的速度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过将eBPF替代iptables，能有效改善这种情况。CNI插件Calico和Cilium尤其醉心于此。下面我们以Calico来看看它是如何利用eBPF来替代iptables的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从网络的角度来看，我们使用eBPF主要是为了两个目的：packet capturing 和 filtering。这表示应用程序可以在网络包流经路径上插入各种eBPF prog以便来抓取数据包的信息并对特定的网络包进行各种操作。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Networking data path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在谈到eBPF如何替代iptable之前，先让我们来看下网络数据路径的概念。如图3所示，当网络设备驱动收到一个网络包后，XDP会得到最早的机会来接触这个package，此时它操作的数据结构是xdp_md。XDP全名为eXpress Data Path。我觉得比较好的翻译应该是“快速数据路径”，此处的“快速”作何解释呢？在图3中，我特地画出了一条XDP_TX的路径，可以看到当满足特定条件时，它完全避开了tc和协议栈，直接将数据快速地处理掉。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当XDP决定将数据包送往内核做后续处理后，网络中断处理程序会申请skb_buff，接下来traffic control(tc)便开始了它的处理流程，也就是我们听说过的QoS和Queue Descipline。","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意：从这里开始，tc和内核栈以及其它网络内核模块都会以skb_buff为处理对象。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之后，skb_buff向上流入Networking stack，如果一路畅通，最终会进入应用层。图中也同样画出了当应用层向外发送一个数据的时候，所流经的data path。还记得我们上面的河道比喻吗？网络数据包确实如河水一样，在河道里面流淌。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当然这个过程中，iptables依旧位于Networking stack中，我们也没有必要绕开它，只要不设置过多的iptables rule，便可以快速地穿过iptables这道屏障。","attrs":{}}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"接收数据data path：device driver --> xdp -- >tc(ingress) --> networking stack --> socket --> application","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"发送数据data path：application --> socket --> networking stack --> tc(egress) --> device driver","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e7/e7fccac0d0448c5bbc632766538bec4d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图 3：networking data path关键节点示意图","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"介绍完网络数据路径再来看图4。别忘了eBPF里面的字母'F'代表的是Filter。聪明的内核工程师自然是不忘初心，允许我们在网络数据路径若干个关键节点上hook eBPF来过滤网络数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/83/832456eaab57a83b30bc727ac4dd0cb3.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图4：eBPF 在data path上可以hook的各个关键节点示意图（重点是右侧部分，暂时忽略左侧）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图4右侧部分，从下往上可供hook的eBPF类型至少有如下几种：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XDP","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"tc","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"socket filter","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kprobe","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tracepoint","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些hook点可以和图3浅绿色的框中所示的关键节点联系起来一起看。实际上，可供hook的点还有很多。嗯，老规矩，以后慢慢聊，好吧，我承认，其实是好多我也不会，等我学完一阵子后再来卖。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"calico tc eBPF示例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"铺垫了这么多，终于到了介绍该如何利用eBPF来实现Network Policy的时候了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图是一张利用eBPF hook到tc来实现Network Policy的架构图。图中eBPF prog hook在与Pod相连的veth上，它包括3大主要的子program：main prog, policy prog和 epilogue prog。利用eBPF的tail call功能，这3个prog依次被调用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图中eBPF prog会接收到来自物理网卡和节点上其它虚拟设备发过来的traffic。而我们看到policy prog自然地会想到Network Policy。没错，通过将Network Policy转译成这里需要的命令，即可方便、快速地控制traffic是否可以流向Pod，而这个过程中我们可以看到iptables被完美地避开了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"强调一下，这里所说的避开不是说流量不通过iptables（实际上节点上其它虚拟设备发过来的traffic可能不可避免地还是会通过iptables过滤一次），而是说因为有了tc eBPF的存在，我们便可以不再依赖iptables，不需要创建巨量的iptables rule，从而显著减低iptables带来的性能影响。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7a/7a8e16617b1399957dd04c978d619a6e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图5：CNI calico利用eBPF来控制traffic示意图","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这张图里面的policy prog会引用到一个IP set map。聪明的你一定会想到可以从user space把允许访问这个Pod的IP和拒绝访问的IP做成allow list和deny list，然后塞到这个map里，而policy prog可以根据你的设置来决定是否对traffic放行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"完美的实现！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是本文的全部内容。码字不易，更多内容请关注二哥的微信公众号。您的举手之劳是对二哥莫大的鼓励。感谢有你！","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/63/63b236f9f533c70d2e68036f810d0391.jpeg","alt":null,"title":"","style":[{"key":"width","value":"25%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}