【得物技術】交易軌跡系統

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"我們不生產數據,我們只是數據的搬運工。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現有系統操作日誌都分散在各個域的系統中,而且沒有形成規範的系統。導致排查線上問題全是去日誌系統各種搜,效率很低。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們針對系統調用鏈路有trace串聯,跨域查問題很爽。但是我們對單據緯度沒有trace串聯,單據流轉對於我們來說是黑盒。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開發人員對自己的功能線上情況不清楚,比如:每天賣家發貨量多少?不清楚;成功率多少?不清楚;平均響應多少?不清楚;針對當前量,我們是不是需要做一些優化?其實有時候並不是不去關注,只是沒有一個很直觀的系統去很直觀的呈現出來。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決上述問題,交易軌跡系統就應運而生了。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"實現思路","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"怎樣去埋數據?","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1. 方案一:手動埋點(不推薦)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對現有系統侵入性較高,而且埋點對系統穩定性有一定的影響,不到萬不得已不推薦。但是我們提供了這項功能。支持三種接入方式,feign、dubbo和rocketmq。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"2. 方案二:日誌清洗(推薦)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"怎樣對現有系統不改造,利用現有的資源去尋找事件action?我們發現access.log可以完美的利用起來。但是有一些超時任務和mq觸發的任務沒有access.log,現階段需要考慮其他方式接入。後續我們考慮提供公共工具,將mq和超時任務觸發的動作,寫入access.log並提供全套的清洗入庫套件。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3. 方案三:binlog(不推薦)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲什麼不推薦,因爲binlog拉取不到入參出參,對排查問題沒有太大的作用。但是有一點優勢很明顯,我們可以追蹤前後兩次的表更。比如:修改地址(A地址->B地址)","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/eb/ebe1b69f529672c447fb0e53a4471ae2.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":4}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"怎樣去數據清洗?","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1. 功能獨立","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和業務系統隔離,獨立一套系統去做數據清洗","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"2. 實時生效","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用groovy腳本做清洗邏輯,腳本寫好後保存實時生效。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3. 數據擴展","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌裏面的數據不滿足埋點需求,我們提供了dubbo和feign的擴展,可以去業務系統查詢數據","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"4. 批量操作支持","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"批量操作會涉及到多個單號的流轉,腳本默認返回List,對多個單號進行埋點,支持一次解析,多條數據落庫","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"5. 主子單號支持","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(同理批量操作)","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6. 日誌太複雜?解析費勁?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對日誌分析,我們提供的公共的數據獲取工具,直接調用方法即可","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例子(寄售申請):","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"package com.shizhuang.duapp.trade.script.biz.deposit.js\n\nimport com.alibaba.fastjson.JSONArray\nimport com.alibaba.fastjson.JSONObject\nimport com.google.common.collect.Lists\nimport com.shizhuang.duapp.trade.cycle.api.resultdata.BizResult\nimport com.shizhuang.duapp.trade.cycle.api.script.AccessLogToBizResultBaseScript\nimport com.shizhuang.duapp.trade.cycle.api.sourcedata.AccessLogData\nimport com.shizhuang.duapp.trade.cycle.api.util.AccessLogUtils\n\n/**\n * @program: trade-cycle-center\n *\n * @author: 小豬佩奇*\n * @create: 2020-11-20 17:08\n * */\nclass ConsignApplyCreate2BizResult extends AccessLogToBizResultBaseScript {\n List parse(AccessLogData accessLogData) {\n // 響應結果\n String responseData = accessLogData.getResponseData()\n // 執行時間\n String takeTime = accessLogData.getTakeTime()\n // http 返回code碼\n String httpStatus = accessLogData.getHttpStatus()\n // 獲取uid\n String uid = AccessLogUtils.getUid(accessLogData)\n List applyNoList = this.analyzeApplyItemNo(responseData);\n List bizResultList = Lists.newArrayList();\n for (applyItemNo in applyNoList) {\n BizResult result = new BizResult()\n result.setOperatorId(uid)\n result.setOperationBizNo(applyItemNo)\n result.setOperationTime(accessLogData.getOperationTime())\n result.setOperationSubType(\"app寄售申請\")\n result.setOperationTrace(accessLogData.getOperationTrace())\n result.setOperationCostTime(Long.valueOf(takeTime))\n result.setOperationResult(\"200\" == httpStatus ? 0 : 1)\n //result.setOperationExtend()\n result.setOperationDetailReq(accessLogData.getPayload())\n result.setOperationDetailResp(accessLogData.getResponseData())\n bizResultList.add(result);\n }\n return bizResultList;\n }\n\n /**\n * 分析單號\n * @param responseData\n * @return\n */\n List analyzeApplyItemNo(String responseData) {\n List list = Lists.newArrayList()\n\n JSONObject jsonObject = JSONObject.parseObject(responseData)\n JSONObject data = jsonObject.getJSONObject(\"data\")\n JSONObject applyProduct = data.getJSONObject(\"applyProduct\")\n if (Objects.isNull(applyProduct)) {\n return list\n }\n\n JSONArray applyItems = applyProduct.getJSONArray(\"applyItems\")\n if (Objects.nonNull(applyItems)) {\n for (Object applyItem : applyItems) {\n JSONObject parseObject = JSONObject.parseObject(applyItem.toString())\n String applyItemNo = parseObject.getString(\"applyItemNo\");\n list.add(applyItemNo);\n }\n }\n return list\n }\n}\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"怎樣去做數據呈現?","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1. 單據緯度","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2c/2c6ae1219909f1d76d45f70f7dcf47c8.png","alt":null,"title":"寄存單交易軌跡","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/71/714cf0e3d3cdeadbcb9bae513bd1df82.png","alt":null,"title":"客服工單軌跡","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查詢時段段內,單據的流轉軌跡(支持跨域跨系統展示)。根據來源數據和結果數據,排查問題根源。如需查看詳細日誌,可以根據traceId去日誌系統去查看詳細trace日誌。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"2. 人緯度","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5f/5f882ef4010c2235349934568d2d600e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統計時間段內用戶的操作行爲,方便排查某個時間段內用戶的操作。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"怎樣自定義圖形化?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TDengine時序數據庫存儲的存儲的數據都是帶有時間戳的,且本身很好地支持時間維度的聚合,正好可以結合grafana對操作日誌中的數據進行分析,展示各類業務運轉的情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/03/03e019bbb8a8ce6b59012d813feff216.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"系統接入","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"access log方式接入","attrs":{}}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"下載腳本項目,編寫腳本代碼","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"新增事件腳本類,將腳本代碼配置到交易軌跡系統","attrs":{}}]}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3a/3a88cf5917aa75531014e958a8fb4e86.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"3","normalizeStart":"3"},"content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"添加路由規則,關聯事件和腳本","attrs":{}}]}],"attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e2/e2c0fec7681bfb574ab1130a63691a32.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"rocketMq方式接入","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"mq接入方式和rpc方式相同,各業務系統把數據封裝好,通過硬編碼的方式將數據發送到mq,軌跡系統訂閱消息進行數據清洗並持久化。有時候獲取事件不是很方便,通過mq的方式接入也未嘗不是一種好的方式。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"擴展","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在存儲上我們支持跨度一年的軌跡查詢,在存儲上已經做過優化,接入方可以不考慮存儲問題。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在和財務系統、客服系統的對接中。我們發現還有更多的方向去擴展功能,不僅僅是侷限於交易軌跡。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"財務系統需要記錄和第三方交互的請求報文,排查問題的時候直接通過單號快速查詢第三方報文。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客服系統需要記錄客服會話的生命週期,通過會話id可以直觀的查看會話鏈路。並且可以通過客服id可以快速統計客服的接入量統計。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後續會加強圖形化的功能和簡易數據分析的能力","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"理論上我們只提供了一種存儲能力,規範了數據存儲的格式。具體怎樣使用,使用方可以自定義。","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後打個廣告,歡迎大家接入。","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文/得物技術 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Ambition","attrs":{}}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章