BPF 和 Go: Linux 中的现代内省形式

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文将向你介绍为什么我们需要像BPF这样的东西，并帮助你了解何时及如何使用它，以及它是如何帮助作为工程师的你改进你正在进行的项目的。我们还将研究它与Go相关的一些详细信息。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/54\/54f298885f5b4869cd408cdd5ae3befe.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"每个人都有自己最喜欢的魔术。对一个人来说他是托尔金（Tolkien），对另一个人来说是普拉切特（Pratchett），对于第三个人来说，比如我，是马克斯·弗雷（Max Frei）。今天我要给大家讲的是我最喜欢的IT魔术：BPF以及围绕它的现代基础设施。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/zh.wikipedia.org\/zh-hans\/BPF","title":"xxx","type":null},"content":[{"type":"text","text":"BPF"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前正处于流行的高峰期。这项技术正在飞速发展，深入到了意想不到的领域，并且越来越容易被普通用户所接受。现在几乎每个流行的会议都有关于这个主题的演讲，早在8月份，我就应邀在GopherCon Russia上做了该主题相关的演讲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我在那里有过非常好的体验，所以我想与尽可能多的人分享一下。本文将向你介绍为什么我们需要像BPF这样的东西，并帮助你了解何时及如何使用它，以及它是如何帮助作为工程师的你改进你正在进行的项目的。我们还将研究它与Go相关的一些详细信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我真正希望的是，你读完这篇文章后，就像第一次读完《哈利波特》的小孩儿那样，眼睛里闪着光芒，并且希望你能够亲自去尝试一下这个新“玩具”。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一些背景知识"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好吧，让一个34岁、留着大胡子、眼神灼热的家伙来告诉你这个魔术是什么？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我们生活在2020年。打开推特，你可以看到愤怒的技术人员发来的推文，他们都说今天编写的软件质量太差了，需要扔掉，我们需要重新开始。有些人甚至威胁说要彻底离开这个行业，因为他们无法忍受这些，一切都是如此的糟糕、不方便且缓慢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d0\/d04fd1ba87593cc0b853ad3cd736d24b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"他们可能是对的：如果不阅读上千条评论，就无法确定原因。但有一点我绝对同意，那就是现代软件堆栈比以往任何时候都要复杂：我们有BIOS、EFI、操作系统、驱动程序、模块、库、网络交互、数据库、缓存、编排器（如K8s）、Docker容器，最后还有我们自己带有运行时和垃圾收集器的软件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一个真正的专业人士可能会花上几天的时间才能回答这样一个问题：当在你的浏览器中输入google.com后会发生什么。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"要理解你的系统发生了什么是非常复杂的，尤其是在当前情况下，出现了问题，你正在赔钱的时候。正是由于这个问题，才出现了能够帮助你了解系统内部情况的企业。在大公司里，有的整个部门都是像夏洛克·福尔摩斯（Sherlock holmes）那样的侦探，他们知道在哪里敲敲锤子，知道用什么拧紧螺栓以节省数百万美元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我喜欢问人们如何在最短的时间内调试出突发问题。通常，人们最先想到的方法是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"分析日志"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。但问题在于，唯一可访问的日志是开发人员放在他们的系统中的日志，这是很不灵活的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第二种最流行的方法是"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"研究度量指标"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。最流行的三个度量指标处理系统都是用Go编写的。度量指标非常有用，但是，虽然它们确实可以让你看到症状，但它们并不总是能够帮助你定义出问题的根本原因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第三种方法是所谓的“"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"可观察性"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"”：你可以对系统的行为提出尽可能多的复杂问题，并获得这些问题的答案。由于问题可能会非常复杂，所以答案可能会需要最广泛的信息，而在问题被提出之前，我们并不知道这些信息是什么，这意味着可观察性绝对需要灵活性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"提供一个“动态”更改日志级别的机会怎么样呢？如果使用调试器，在程序运行时连接到程序，并在不中断程序运行时执行某些操作，又会怎么样呢？了解哪些查询会被发送到系统中，可视化慢查询的来源，通过pprof查看内存消耗情况，并获取其随时间变化的曲线图呢？如何测量一个函数的延迟以及延迟对参数的依赖性呢？我将所有这些方法都归为“可观测性”这一总称。这是一套实用程序、方法、知识和经验，它们结合在一起，共同为我们提供了机会，即使不能做到我们想做的任何事，至少在系统工作时，它可以在系统中做很多“现场”工作。它相当于现代IT界的一把瑞士军刀。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d5\/d50afde5664288212699ada344b3ba2b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"但是我们怎样才能做到这一点呢？市场上已经有很多现成的工具了：有简单的、有复杂的、有危险的、也有缓慢的。 但是今天这篇文章是关于BPF的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Linux内核是一个事件驱动系统。实际上，在内核以及整个系统中所发生的一切都可以看作是一组事件。中断是一个事件；通过网络接收数据包是一个事件；将处理器的控制权转移到另一个进程是一个事件；运行函数也是一个事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"是的，所以BPF是Linux内核的一个子系统，它使你有机会编写一些由内核运行以响应事件的小程序。这些程序既可以帮忙你了解系统正在发生什么，也可以用来控制系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"现在让我们来深入了解一下详细细节。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"什么是eBPF？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF的第一个版本于1994年问世。有些人在为tcpdump实用程序编写用于查看或“嗅探”网络数据包的简单规则时，可能会遇到过它。你可以为tcpdump设置过滤器，这样你就不必查看所有的内容，只需查看你感兴趣的数据包即可。例如，“只查看TCP协议和80端口”。对于每个传递的数据包，都会运行一个函数来确定其是否需要保存有问题的特定数据包。可能会有很多数据包，所以我们的函数必须要很快。实际上，我们的tcpdump过滤器被转换为BPF函数。下面是一个例子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3b\/3bb4bb9dbee19be46a82e34db8b9edb5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#757575","name":"user"}}],"text":"一个简单的以BPF程序形式呈现的tcpdump过滤器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"最初的BPF代表了一个非常简单带有多个寄存器的虚拟机。但是，尽管如此，BPF还是大大加快了网络数据包的过滤速度。在当时，这是一个很重要的进步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/41\/41f17bc24c3f33603223fe77b8b682b8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在2014年，"},{"type":"link","attrs":{"href":"https:\/\/www.linkedin.com\/in\/alexey1\/","title":null,"type":null},"content":[{"type":"text","text":"Alexei Starovoitov"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，一个非常著名的内核黑客，扩展了BPF的功能。他增加了寄存器的数量和程序允许的大小，添加了JIT编译，并创建了一个用于检查程序是否安全的检查器。然而，最令人印象深刻的是，新的BPF程序不仅能够在处理数据包时运行，而且还能够响应其他内核事件，并能在内核和用户空间之间来回传递信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这些变化为使用BPF的新方法提供了机会。一些过去需要通过编写复杂而危险的内核模块来实现的功能，现在可以相对简单地通过BPF来实现。为什么能这么好呢？是因为在编写模块时，任何错误通常都会导致死机（panic），不是“温和”Go风格的死机，而是内核死机，一旦发生，我们唯一能做的就是重启。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"普通的Linux用户突然拥有了一项新的超能力：能够查看“引擎盖下”的情况——这是以前只有核心内核开发人员才能使用的东西，或者根本不会提供给任何人。这个选项可以与为iOS或Android编写程序的能力相媲美：在旧手机上，这要么是不可能的，要么就是要复杂得多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Alexei Starovoitov新版本的BPF被称为eBPF（e代表扩展，extended）。但是现在它已经取代了所有旧的BPF用法，并且已经变得非常流行了，为了简单起见，它仍然被称为BPF。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF可用于何处？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好了，你可以将BPF程序附加到哪些事件或触发器上呢，人们又是如何开始使用它们以获取新的能力的呢?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前，主要有两大组触发器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第一组用于处理网络数据包和管理网络流量。它们是XDP、流量控制事件及其他几个事件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"以下情况需要用到这些事件："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"创建简单但非常有效的防火墙。Cloudflare和Facebook等公司使用BPF程序来过滤掉大量的寄生流量，并打击最大规模的DDoS攻击。由于处理发生在数据包生命的最早阶段，直接在内核中进行（BPF程序的处理有时甚至可以直接推送到网卡中进行），因此可以通过这种方式处理巨量的流量。这些事情过去都是在专门的网络硬件上完成的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"创建更智能、更有针对性、但性能更好的防火墙——这些防火墙可以检查通过的流量是否符合公司的规则、是否存在漏洞模式等。例如，Facebook在内部进行这种审计，而一些项目则对外销售这类产品。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"创建智能负载均衡器。最突出的例子就是Cilium项目，它最常被用作K8s集群中的网格网络。Cilium对流量进行管理、均衡、重定向和分析。所有这些都是在内核运行的小型BPF程序的帮助下完成的，以响应这个或那个与网络数据包或套接字相关的事件。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这是第一组与网络问题相关并能够影响网络通信行为的触发器。 第二组则与更普遍的可观察性相关；在大多数情况下，这组的程序无法影响任何事件，而只能“观察”。这才是我更感兴趣的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这组的触发器有如下几个："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"perf事件（perf events）——与性能和perf Linux分析器相关的事件：硬件处理器计数器、中断处理、小\/大内存异常拦截等等。例如，我们可以设置一个处理程序，每当内核需要从swap读取内存页时，该处理程序就会运行。例如，想象有这样一个实用程序，它显示了当前所有使用swap的程序。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"跟踪点（tracepoints）——内核源代码中的静态（由开发人员定义）位置，通过附加到这些位置，你可以从中提取静态信息（开发人员先前准备的信息）。在这种情况下，静态似乎是一件坏事，因为我说过，日志的缺点之一就是它们只包含了程序员最初放在那里的内容。从某种意义上说，这是正确的，但跟踪点有三个重要的优势："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有相当多的跟踪点散落在内核中最有趣的地方"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"当它们不“开启”时，它们不使用任何资源"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"它们是API的一部分，它们是稳定的，不会改变。这非常重要，因为我们将提到的其他触发器缺少稳定的API。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"例如，假设有一个关于显示的实用程序，内核出于某种原因没有给它时间执行。 你坐着纳闷为什么它这么慢，而pprof却没有显示任何什么有趣的东西。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"USDT——与跟踪点相同，但是它适用于用户空间的程序。也就是说，作为程序员，你可以将这些位置添加到你的程序中。并且许多大型且知名的程序和编程语言都已经采用了这些跟踪方法：例如MySQL、或者PHP和Python语言。通常，它们的默认设置为“关闭”，如果要打开它们，需要使用enable-dtrace参数或类似的参数来重新构建解释器。是的，我们还可以在Go中注册这种类跟踪。你可能已经识别出参数名称中的单词"},{"type":"link","attrs":{"href":"https:\/\/en.wikipedia.org\/wiki\/DTrace","title":null,"type":null},"content":[{"type":"text","text":"DTrace"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。关键在于，这些类型的静态跟踪是由"},{"type":"link","attrs":{"href":"https:\/\/en.wikipedia.org\/wiki\/Solaris_(operating_system)","title":null,"type":null},"content":[{"type":"text","text":"Solaris"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"操作系统中诞生的同名系统所推广的。例如，想象一下，何时创建新线程、何时启动GC或与特定语言或系统相关的其他内容，我们都能够知道是怎样的一种场景。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这是另一种魔法开始的地方："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Ftrace触发器为我们提供了在内核的任何函数开始时运行BPF程序的选项。这是完全动态的。这意味着内核将在你选择的任何内核函数或者在所有内核函数开始执行之前，开始执行之前调用你的BPF函数。你可以连接到所有内核函数，并在输出时获取所有调用的有吸引力的可视化效果。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"kprobes\/uprobes提供的功能与ftrace几乎相同，但在内核和用户空间中执行函数时，你可以选择将其附加到任何位置上。如果在函数的中间，变量上有一个“if”，并且能为这个变量建立一个值的直方图，那就不是问题。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"kretprobes\/uretprobes——这里的一切都类似于前面的触发器，但是它们可以在内核函数或用户空间中的函数返回时触发。这类触发器便于查看函数的返回内容以及测量执行所需的时间。例如，你可以找出“fork”系统调用返回的PID。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我再重复一遍，所有这些最奇妙之处在于，当我们的BPF程序为了响应这些触发器而被调用之后，我们可以很好地“环顾四周”：读取函数的参数，记录时间，读取变量，读取全局变量，进行堆栈跟踪，保存一些内容以备后用，将数据发送到用户空间进行处理，和\/或从用户空间获取数据或一些其他控制命令以进行过滤。简直不可思议！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我不知道你是怎么想的，但对我来说，这个新的基础设施就像是一个我很早之间就想要得到的玩具一样。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"API：怎么使用它"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"好了，让我们开看一下BPF程序由什么组成的，以及如何与它交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/63\/636083ff04253a97027a107136a96aa2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"首先，我们有一个BPF程序，如果它通过验证，就会被加载到内核中。在那里，它将被JIT编译器编译成机器码，并在内核模式下运行，这时附加的触发器将会被激活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BPF程序可以选择与第二部分交互，即与用户空间程序交互。有两种方式可以做到这一点。我们可以向循环缓冲区写入，而用户空间程序可以从中读取。我们也可以对键值映射（也就是所谓BPF映射）进行写入和读取，相应地，用户空间程序也可以做同样的事情，并且相应地，它们就可以相互传递信息了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基本用法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"使用BPF最简单的方法是用C语言编写BPF程序，然后用Clang编译器将相关的代码编译成虚拟机的代码（在任何情况下都不应该采用这种从头开始的方式）。然后，我们直接使用BPF系统调用加载该代码，并同样采用BPF系统调用的方式与我们的BPF程序进行交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"第一种可用的简化方法是使用libbpf库。它是随内核源代码一起提供的，允许我们直接使用BPF系统调用。基本上，它为加载代码和使用BPF映射将数据从内核发送到用户空间并返回提供了方便的包装器。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/iovisor\/bcc","title":null,"type":null},"content":[{"type":"text","text":"bcc"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"显然，这对人们来说一点也不方便。幸运的是，在iovizor这个品牌下，BCC项目出现了，这使我们的生活变得更加轻松了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b5\/b557c5545784caf46bec5122f1605d1f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基本上，它为我们准备了整个构建环境，并允许我们编写单个的BPF程序，其中С语言部分会被自动构建并加载到内核中，而用户空间部分则可以用Python来实现，这既简单又清晰明了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/iovisor\/bpftrace","title":null,"type":null},"content":[{"type":"text","text":"bpftrace"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"然而，BCC似乎在很多方面都很复杂。出于某些原因，人们特别不喜欢用С语言来写内核的这部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"同样那些来自iovizor的人也提供了一个工具，bpftrace，它允许我们用类似于AWK这样的简单脚本语言（甚至是单行代码）来编写BPF脚本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fb\/fbac146c7db1807a05ef8aa16c1c7963.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Brendan Gregg是生产力和可观察性领域的知名专家，他对BPF的可用工作方式进行了可视化，如下图所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/02\/02b545daec8f85c209a839a7759abf35.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"纵轴显示的是给定工具的易用性，而横轴显示则是其功能。我们可以看到，BCC是一个非常强大的工具，但它并不是一个超级简单的工具。而bpftrace要简单得多，但同时，它的功能则稍弱一些。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"使用BPF的示例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"现在，让我们来看一些具体的例子，看看这些我们可以利用的神奇的力量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"BCC和bpftrace都包含了一个“工具”目录，其中包含了大量的有趣且有用的现成脚本。它们也可以用作本地的Stack Overflow，你可以从中复制代码块以用于自己的脚本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"例如，下面是一个显示DNS查询延迟的脚本："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─marko@marko-home ~\n╰─$ sudo gethostlatency-bpfcc\nTIME PID COMM LATms HOST\n16:27:32 21417 DNS Res~ver #93 3.97 live.github.com\n16:27:33 22055 cupsd 7.28 NPI86DDEE.local\n16:27:33 15580 DNS Res~ver #87 0.40 github.githubassets.com\n16:27:33 15777 DNS Res~ver #89 0.54 github.githubassets.com\n16:27:33 21417 DNS Res~ver #93 0.35 live.github.com\n16:27:42 15580 DNS Res~ver #87 5.61 ac.duckduckgo.com\n16:27:42 15777 DNS Res~ver #89 3.81 www.facebook.com\n16:27:42 15777 DNS Res~ver #89 3.76 tech.badoo.com :-)\n16:27:43 21417 DNS Res~ver #93 3.89 static.xx.fbcdn.net\n16:27:43 15580 DNS Res~ver #87 3.76 scontent-frt3-2.xx.fbcdn.net\n16:27:43 15777 DNS Res~ver #89 3.50 scontent-frx5-1.xx.fbcdn.net\n16:27:43 21417 DNS Res~ver #93 4.98 scontent-frt3-1.xx.fbcdn.net\n16:27:44 15580 DNS Res~ver #87 5.53 edge-chat.facebook.com\n16:27:44 15777 DNS Res~ver #89 0.24 edge-chat.facebook.com\n16:27:44 22099 cupsd 7.28 NPI86DDEE.local\n16:27:45 15580 DNS Res~ver #87 3.85 safebrowsing.googleapis.com\n^C%"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这是一个实时显示DNS查询完成时间的实用程序，因此，你可以捕获一些意外的异常值（举个例子）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"如下则是一个“监视”别人在终端上输入的内容的脚本："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─marko@marko-home ~\n╰─$ sudo bashreadline-bpfcc \nTIME PID COMMAND\n16:51:42 24309 uname -a\n16:52:03 24309 rm -rf src\/badoo"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这种脚本可以用来捕获“坏邻居”，或者对公司的服务器执行安全审计。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用于查看高级语言调用流的脚本如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"╭─marko@marko-home ~\/tmp\n╰─$ sudo \/usr\/sbin\/lib\/uflow -l python 20590\nTracing method calls in python process 20590... Ctrl-C to quit.\nCPU PID TID TIME(us) METHOD\n5 20590 20590 0.173 -> helloworld.py.hello \n5 20590 20590 0.173 -> helloworld.py.world \n5 20590 20590 0.173 helloworld.py.hello \n5 20590 20590 1.174 -> helloworld.py.world \n5 20590 20590 1.174 helloworld.py.hello \n5 20590 20590 2.176 -> helloworld.py.world \n5 20590 20590 2.176 helloworld.py.hello \n6 20590 20590 3.176 -> helloworld.py.world \n6 20590 20590 3.176 helloworld.py.hello \n6 20590 20590 4.177 -> helloworld.py.world \n6 20590 20590 4.177