go-zero流数据处理利器

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流处理 (Stream processing) 是一种计算机编程范式，其允许给定一个数据序列 (流处理数据源)，一系列数据操作 (函数) 被应用到流中的每个元素。同时流处理工具可以显著提高程序员的开发效率，允许他们编写有效、干净和简洁的代码。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流数据处理在我们的日常工作中非常常见，举个例子，我们在业务开发中往往会记录许多业务日志，这些日志一般是先发送到 Kafka，然后再由 Job 消费 Kafaka 写到 elasticsearch，在进行日志流处理的过程中，往往还会对日志做一些处理，比如过滤无效的日志，做一些计算以及重新组合日志等等，示意图如下:"},{"type":"link","attrs":{"href":"https://static.gocn.vip/photo/2020/c86171b0-a3e0-4565-a2a1-fd65b07a1da4.png?x-oss-process=image/resize,w_800","title":null}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8e3aced638356acfa73c88b5cfb4fbd8.png","alt":"fx_log","title":"","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"流处理工具 fx"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/tal-tech/go-zero","title":""},"content":[{"type":"text","text":"go-zero"}]},{"type":"text","text":"是一个功能完备的微服务框架，框架中内置了很多非常实用的工具，其中就包含流数据处理工具"},{"type":"link","attrs":{"href":"https://github.com/tal-tech/go-zero/tree/master/core/fx","title":""},"content":[{"type":"text","text":"fx"}]},{"type":"text","text":"，下面我们通过一个简单的例子来认识下该工具："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"package main\n\nimport (\n \"fmt\"\n \"os\"\n \"os/signal\"\n \"syscall\"\n \"time\"\n\n \"github.com/tal-tech/go-zero/core/fx\"\n)\n\nfunc main() {\n ch := make(chan int)\n\n go inputStream(ch)\n go outputStream(ch)\n\n c := make(chan os.Signal, 1)\n signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)\n = 0; i-- {\n opp := len(items) - 1 - i\n items[i], items[opp] = items[opp], items[i]\n }\n\n // 写入流\n return Just(items...)\n}\n"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Distinct"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"distinct 对流中元素进行去重，去重在业务开发中比较常用，经常需要对用户 id 等做去重操作:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"// 例子\nfx.Just(1, 2, 2, 2, 3, 3, 4, 5, 6).Distinct(func(item interface{}) interface{} {\n return item\n}).ForEach(func(item interface{}) {\n fmt.Println(item)\n})\n// 结果为 1，2，3，4，5，6\n\n// 源码\nfunc (p Stream) Distinct(fn KeyFunc) Stream {\n source := make(chan interface{})\n\n threading.GoSafe(func() {\n defer close(source)\n // 通过key进行去重，相同key只保留一个\n keys := make(map[interface{}]lang.PlaceholderType)\n for item := range p.source {\n key := fn(item)\n // key存在则不保留\n if _, ok := keys[key]; !ok {\n source