go-zero流數據處理利器

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流處理 (Stream processing) 是一種計算機編程範式，其允許給定一個數據序列 (流處理數據源)，一系列數據操作 (函數) 被應用到流中的每個元素。同時流處理工具可以顯著提高程序員的開發效率，允許他們編寫有效、乾淨和簡潔的代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流數據處理在我們的日常工作中非常常見，舉個例子，我們在業務開發中往往會記錄許多業務日誌，這些日誌一般是先發送到 Kafka，然後再由 Job 消費 Kafaka 寫到 elasticsearch，在進行日誌流處理的過程中，往往還會對日誌做一些處理，比如過濾無效的日誌，做一些計算以及重新組合日誌等等，示意圖如下:"},{"type":"link","attrs":{"href":"https://static.gocn.vip/photo/2020/c86171b0-a3e0-4565-a2a1-fd65b07a1da4.png?x-oss-process=image/resize,w_800","title":null}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8e3aced638356acfa73c88b5cfb4fbd8.png","alt":"fx_log","title":"","style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"流處理工具 fx"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/tal-tech/go-zero","title":""},"content":[{"type":"text","text":"go-zero"}]},{"type":"text","text":"是一個功能完備的微服務框架，框架中內置了很多非常實用的工具，其中就包含流數據處理工具"},{"type":"link","attrs":{"href":"https://github.com/tal-tech/go-zero/tree/master/core/fx","title":""},"content":[{"type":"text","text":"fx"}]},{"type":"text","text":"，下面我們通過一個簡單的例子來認識下該工具："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"package main\n\nimport (\n \"fmt\"\n \"os\"\n \"os/signal\"\n \"syscall\"\n \"time\"\n\n \"github.com/tal-tech/go-zero/core/fx\"\n)\n\nfunc main() {\n ch := make(chan int)\n\n go inputStream(ch)\n go outputStream(ch)\n\n c := make(chan os.Signal, 1)\n signal.Notify(c, syscall.SIGTERM, syscall.SIGINT)\n = 0; i-- {\n opp := len(items) - 1 - i\n items[i], items[opp] = items[opp], items[i]\n }\n\n // 寫入流\n return Just(items...)\n}\n"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Distinct"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"distinct 對流中元素進行去重，去重在業務開發中比較常用，經常需要對用戶 id 等做去重操作:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"go"},"content":[{"type":"text","text":"// 例子\nfx.Just(1, 2, 2, 2, 3, 3, 4, 5, 6).Distinct(func(item interface{}) interface{} {\n return item\n}).ForEach(func(item interface{}) {\n fmt.Println(item)\n})\n// 結果爲 1，2，3，4，5，6\n\n// 源碼\nfunc (p Stream) Distinct(fn KeyFunc) Stream {\n source := make(chan interface{})\n\n threading.GoSafe(func() {\n defer close(source)\n // 通過key進行去重，相同key只保留一個\n keys := make(map[interface{}]lang.PlaceholderType)\n for item := range p.source {\n key := fn(item)\n // key存在則不保留\n if _, ok := keys[key]; !ok {\n source