從0開始構建核心業務微服務治理平臺的實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"導讀"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"近年來，FreeWheel核心業務開發團隊致力於將傳統單體Rails應用，向分佈式微服務架構遷移，以適應越來越複雜的業務場景和系統性能的提升。隨着微服務規模的不斷增長，一些新的問題也隨之產生。其中如何對這些業務服務進行有效的治理和維護，對業務狀態進行監控，甚至於線上調試變得尤爲重要。業務服務治理平臺（business service management platform），是我們爲應對這一挑戰做出的選擇。本文將詳細解析FreeWheel核心業務開發團隊構建的服務治理平臺。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"爲什麼需要服務治理平臺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"隨着Rails單體應用向分佈式微服務架構遷移的深入，面向不同業務和層次的微服務如雨後春筍般誕生，微服務集羣的規模迅速增長。架構遷移讓我們可以把業務重新梳理、聚合和解藕，不同的微服務可以聚焦自身業務，自成體系進行維護，減少了對其他業務的影響，增加了系統整體的可擴展性。但同時，這也導致有越來越多的微服務需要治理，原本只需要對一個單體應用進行監控管理，如今需要對幾十個甚至上百個微服務進行管理。另一方面，分佈式架構本身也會引入一些在單體應用時期所不存在的問題，例如分佈式事務、消息等，對於這些方面我們也缺少相關的治理平臺。因此，在我們的分佈式微服務實踐過程中，經常需要面對以下這些問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"微服務在出錯或響應慢時，如何能進行簡單快速的調試，以便了解是微服務本身的問題，還是所依賴的服務有問題？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"相比於單體應用，分佈式系統更容易引入數據不一致，如何對這樣的數據進行監控？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在基於異步消息的業務中，某個主題的業務沒能正常完成，是生產者沒有把消息發出來？還是消費者沒有接收到消息？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爲什麼數據庫已經更新的數據遲遲沒有生效？緩存數據何時過期？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我們有哪些後臺任務正在執行？執行的排期如何？執行失敗的原因是什麼？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"等等……"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"微服務的部署層面我們可以通過業界雲原生的解決方案來實現，而這些業務層面的問題卻很難找到一個公共的解決方案或者治理入口，因爲這往往是跟具體的業務場景深度綁定的。不同的微服務業務場景和功能不同，但基本都遵從類似的開發實踐，使用相同的基礎組件，這使得我們對微服務進行治理成爲可能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"需要注意的是，這裏我們關注的重點是業務，而不是基礎組件，例如我們不關注異步消息機制或組件（Kafka）本身，我們信任基礎組件是正常工作的，而關注基於它所實現的某些業務上的問題。在這些問題解決之前，工程師往往需要耗費巨大的精力在線上環境進行調試、監控或處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"基於此，我們決定從0開始構建一套適用於FreeWheel自己業務場景的服務治理平臺，來對分佈式微服務進行業務治理，解決工程師的痛點。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"業務微服務治理平臺實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"總體來看，我們構建的服務治理平臺有以下幾個方面需要考慮："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"業務服務治理平臺在某些功能上，需要和FreeWheel特有的業務深度綁定。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"避免與FreeWheel已有的PQM（FreeWheel監控平臺），FOC（FreeWheel運維中心）等模塊功能重合。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"平臺應輕量級，能快速迭代開發。當有新需求出現時，可以更快更好地對平臺進行橫向擴展。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"根據上述提到的應用場景，我們需要構建的是一套基於Web UI的業務服務治理平臺。它需要和業務微服務通信，和服務集羣中的公共資源進行整合，幷包含許多業務監控和治理等功能，因此它應該位於微服務集羣中所有微服務之上，如下圖所示。這套平臺被命名爲Falcon，可以直接在生產環境中使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c3\/c3a771e29e6e2689a226a8772ace04d4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"技術選型與架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"針對服務平臺Falcon的構建，我們從以下幾個方面進行了技術對比和選型："}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

Topic	Solution	Pros	Cons	Result
前端技術棧	React	與當前團隊內的實踐一致 非常成熟的社區和生態 今天它仍然在快速發展		✅
平臺編程語言	Ruby on Rails	可以簡單快速搭建Web應用 團隊內有Rails開發經驗和專家資源	在未來可能會碰到性能瓶頸，就像今天的業務應用 相關技術在今天發展緩慢
	Golang	很好的運行時性能 最近幾年生態發展迅猛 技術棧和團隊的微服務實踐一致	代碼比較冗餘 GRPC接口數據轉換實現繁瑣
	Nodejs	可以和前端統一語言於JS 生態有豐富的包可以使用 JSON格式和接口處理方便	未來可能也會有性能瓶頸	✅
數據庫類型	Postgresql	在SQL和NoSQL上豐富的功能和特色支持 相對較高的性能	對工程師和DBA來說相對陌生 我們可能不會使用太多的新功能
	Mysql	對開發團隊簡單友好，基本的需求都能滿足 DBA可以提供更多支持	爲了一些json格式支持等功能，必須使用5.7及以上版本	✅
數據庫運行時	K8S pod\/pv\/pvc	很輕量級 Kubernetes Pod資源已經存在，可以節省開銷	用Pod的形式比較難維護 需要在集羣中開放額外的路由以便從外網訪問
	AWS RDS	DBA可以提供維護，工程師可以專注於使用 請求量超過1M\/月時纔會收費，廉價 AWS雲原生，周邊基礎設施支持齊全	在現階段對於一個內部使用的Web平臺而言略重	✅"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"根據以上對比，基於對治理平臺快速開發、穩定運行的要求，我們最終選用React+Nodejs+Mysql+AWS RDS的組合來進行開發構建。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"按照FreeWheel今天的業務場景需求，我們的治理平臺將包括四個主要的部分：Falcon前端，Falcon後端，數據庫，Redis。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爲了與現有的服務集羣進行整合，我們需要將治理平臺的各模塊部署在AWS集羣中："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cd\/cd5ff1b9a0d429f22b6fe7dace55f09e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"其中："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon前端主要提供所有的Web前端資源和邏輯，它是獨立於後端進行開發和部署的，實現了前後端的分離解耦。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon後端是整個平臺的後端服務器，負責平臺的所有業務邏輯。同時它需要和集羣中的業務微服務和公共組件進行通信交互。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"數據庫用於存儲平臺自身的數據，例如登錄用戶的信息、採集到的業務數據等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Redis模塊是爲了實現定時任務等功能點所引入的模塊。爲了儘量減少對線上微服務的影響，我們沒有使用集羣中業務微服務所使用的Redis，而是重新部署了一個單獨的Redis。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"我們將Falcon前端\/Falcon後端\/Redis打包，以Kubernetes Pod的形式運行部署，和FreeWheel的業務微服務部署在同一個AWS EKS集羣中，而數據庫使用了AWS RDS服務來支持。我們對Falcon前端\/Falcon後端\/數據庫暴露了VIP，FreeWheel內網進來的請求，會先經過AWS ALB，轉發到服務網格Ingress Gateway，最後打到Falcon對應的Pod或數據庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon與EKS集羣的交互主要通過Falcon後端完成：訂閱監聽Kafka傳遞的消息；搜索讀取集羣Redis中的數據；對業務微服務進行接口調用等。其中，Kafka是FreeWheel使用的分佈式消息發佈訂閱系統，用來傳遞業務微服務之間的異步消息；Redis用於緩存一些不易變的業務數據，或者用於存儲實現後臺任務；業務微服務處理業務請求，會跟AWS S3等設施配合實現業務，使用Aurora存儲業務數據，與FreeWheel後端服務進行交互等。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"平臺運行工作流"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當Falcon被部署運行使用時，會經歷以下過程："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"部署之後，Falcon後端開啓消費者監聽Kafka消息"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon後端加載數據監控配置進Redis，開啓任務調度"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶訪問VIP地址, 請求被路由到Falcon前端Pod，Falcon前端返回JS資源"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"瀏覽器加載並渲染前端資源，進入登錄頁面"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶輸入LDAP用戶名\/密碼，請求路由到Falcon後端，Falcon後端驗證登錄，存儲登錄信息進session和數據庫"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"用戶在平臺界面操作，請求路由到Falcon後端，將操作數據存儲，或實時調用業務微服務，完成對應操作"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/26\/2665d7a23892cb42c55c9c202578c84e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"平臺提供的功能模塊"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"今天Falcon針對於業務團隊的痛點實現了許多功能模塊，還有一些功能模塊正在探索和開發中，下圖是Falcon的導航首頁。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9f\/9ff749e20a889f0249c84ccf2ec23381.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"接下來本文會圍繞Falcon的這些功能模塊進行介紹：登錄驗證，數據監控模塊，後臺任務模塊，異步消息模塊，業務緩存模塊，線上調試模塊，用戶管理模塊和使用記錄模塊。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"準備工作 - 登錄驗證"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"出於系統平臺安全性的考慮，我們限定了只有在FreeWheel內網纔可以訪問Falcon平臺。平臺的使用用戶限定在FreeWheel的工程師團隊，而FreeWheel內部員工使用LDAP來做賬號的統一登錄認證，因此Falcon後端也集成了LDAP對登錄用戶認證。在用戶首次登錄時，Falcon會將該用戶同步存儲在數據庫中，以便之後爲其配置在Falcon平臺的用戶權限。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2e\/2e79f1b7a053f5481c0c1160c97cd46b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"數據監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"數據監控模塊旨在監控異常的業務數據。在從Rails單體應用遷移到分佈式微服務後，很多數據的增刪改不再由原來一個數據庫事務來完成，而是變成了多個微服務多個數據庫事務來進行數據更新，因而很難保證不同微服務間的數據強一致。一個簡單快速的方案就是對業務不一致的數據進行監控，Falcon提供了這樣一個入口進行髒數據的監控和報警，用戶可以通過提供一段SQL語句，或者是微服務實現的一個接口，來達到對特定數據的監控目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a64a5d74a176691265f1a78e0be8f239.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"如上圖所示，Falcon後端在啓動完成後，從數據庫加載數據監控設置，初始化基於Redis的任務隊列。任務隊列中的任務根據設置好的排期，定時調用Handler執行SQL或調用接口，將執行結果寫會到數據庫。同時將執行結果與用戶設置的參數進行比較，一旦發現滿足髒數據條件，即進行報警通知訂閱者。今天Falcon提供了Email和Slack兩種方式進行報警通知。用戶可以實時更改監控設置，Falcon後端會將用戶的實時更改持久化，並更新任務隊列即時生效。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"後臺任務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"後臺任務一般分爲定時任務和按需任務。相信大部分的系統平臺都有與自身業務相關的後臺任務，FreeWheel也不例外。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在Rails單體應用的年代，FreeWheel使用Resque對後臺任務進行管理，Resque自身也提供了一套基於Sinatra的Web管理界面。遷移到Golang微服務之後，FreeWheel使用封裝的bricks\/job作爲後臺任務管理工具，創建、執行、重試、銷燬後臺任務。然而相比於Resque，它對於工程師的痛點在於無法進行可視化的管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"針對這一痛點，我們在Falcon中構建了後臺任務的可視化模塊，提供5個方面的內容：Worker Pool, Queue, Scheduled Job, Retry Job, Dead Job. 用戶可以查看到正在執行的任務有哪些，隊列中已有哪些任務，將要執行的定時任務分別安排在了什麼時間，重新過的任務是哪些，哪些任務執行失敗了等等。例如下圖中展示了Inventory Service中每10秒執行一次的default_check_alive任務的排期情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b6\/b69e160eddfc72c884f7da4568798188.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"特別的，我們可能更關注於哪些任務執行失敗了，以及失敗原因，因此我們把失敗任務的諸如參數、錯誤內容等詳細信息展示出來，並提供了重試功能，以便在工程師在排查完錯誤原因後，可以手動觸發重新執行任務。如下圖展示了Order Service最近執行失敗的任務、任務參數、失敗原因等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5b\/5b7fe8cf18d4e29c7494f912576acaf5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"異步消息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"隨着微服務集羣規模越來越龐大，業務越來越複雜，異步消息機制也被越來越廣泛地應用，微服務在很多業務場景中都需要發送\/接收異步消息。從工程師的角度，我們很希望能實時得知消息是否被成功發送到Kafka，發送的消息內容是否是我們所期望的。以往我們只能通過查看日誌的方式來獲知消息的發送情況，這對工程師是非常不友好的。在這個模塊，我們通過新加消費者監聽消息的方式，並將監聽消息展示，實現了Kafka業務消息的可視化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1e\/1e2d1d6903643c0031f88175fdfb00ca.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon後端啓動完成後，從數據庫加載異步消息監聽配置，最主要的是消息的proto格式文件，通過proto handler將消息格式解析，並啓動初始化consumer監聽該topic消息，將監聽到的消息存儲回數據庫，並構建UI對這些數據進行展示，達到了業務消息可視化的目的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"由於隨着新業務的出現，新的消息也會不斷增加，爲了應對和支持越來越多的新消息，我們也做了動態添加新消息類型並監聽展示的功能。當我們的微服務中增加了新消息，只需要將新消息的topic\/消息的proto定義文件\/消息的名稱配置好，Falcon會將配置實時存入數據庫，並動態加載這個消息並監聽展示。由於消息量巨大，考慮到存儲成本，以及我們對歷史已久的消息並不感興趣，在數據庫層面我們設置了Event Scheduler定期對歷史消息進行清理操作。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"業務緩存"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爲了提升微服務的處理能力和響應性能，減小業務層對數據庫的壓力，我們會在領域微服務中加入緩存，將常用不易變的數據放到緩存中。每次有新的請求過來，先查詢緩存，如果有數據並不過期，則直接讀取返回。類似於後臺任務模塊和異步消息模塊的問題，緩存中存了什麼，有效期多久，何時進行的更新，在微服務運行時我們是無從得知的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一個常見的場景是，數據庫中的數據更新了，卻不能很清楚地知道數據何時能生效，在定位問題時很容易導致判斷錯誤。在這個模塊中，我們將緩存數據進行了可視化展示，提供搜索功能以針對特定的key進行查詢，用戶可以很清楚地看到有哪些數據被緩存，數據量多大，到期時間等等。如下圖展示了Metadata Service中當前緩存的業務數據情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/16\/16a736e4477bc3ecc12be364ea938f39.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"線上調試"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"領域微服務的業務中，往往需要依賴於第三方的服務，而在生產環境中這些第三方服務發生問題時，我們很難快速地從微服務的角度進行問題定位。比如下層服務響應慢，微服務對外的表現也是響應慢，但很難確定是微服務本身操作數據庫慢，還是調用下游服務響應慢。不同的微服務可以根據自己的業務情況，實現自己的調試接口，提供調試信息。線上調試模塊提供了調試入口，將調試接口集成到平臺調試模塊，用戶就可以在平臺手動觸發，查看整個鏈路的執行情況。這在發生線上問題時，能幫助工程師快速定位出錯原因，節約處理時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/00\/00652e143ad3af6793b4e2ee754bb2a8.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" 以Advertising模塊爲例，今天它主要依賴於Feedback\/Counter\/Conflict Detection\/Presto\/S3等服務，我們針對於這些依賴都接入了特定的調試接口。平臺本身提供了快速接入其他接口的代碼模板，在有新的需求出現時能快速擴展。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"用戶管理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon平臺的功能和架構定位，決定了它的重要性和影響程度遠高於微服務。儘管平臺致力於實現對於業務有保護性質的功能，但仍有必要對登錄使用該平臺的用戶進行管理，以避免發生誤操作造成嚴重影響。我們採用了通用的用戶-角色-權限模式來進行用戶的管理，每個功能模塊都定義了讀寫權限，細化到微服務，在此之上按團隊劃分配置了工程師角色，只需要給不同用戶以相應的角色，即對該用戶在平臺的操作權限進行了控制。如下圖是給一個Advertising團隊工程師的權限。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cb\/cbd1edfdeb7f66a6f3d233a588d92e11.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"使用記錄"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"作爲平臺系統完整性的一部分，也爲了更好地追蹤平臺上的設置更改，我們實現了使用記錄模塊，以記錄在該平臺上發生的所有更新操作。由於平臺本身沒有特別的複雜業務，同時更新不會特別頻繁，因而在記使用記錄時我們選擇記錄使用全量，而非變量，即當某個對象發生變化時，都將原始對象的快照進行全量備份。下圖是某次更改設置的新值與舊值的對比，通過記錄全量，我們能很清楚地看到某一時刻整個數據的狀態，也能很容易地看到那些字段發生了變化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7e\/7e7a6e40f725ccabfb6cec53fc724f30.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"結語"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Falcon作爲FreeWheel核心業務開發團隊從0構建的一套微服務治理平臺，提供了諸如數據監控、異步消息等功能模塊，幫助工程師解決了很多在分佈式微服務架構時期所面臨的業務治理或監控痛點。目前這個平臺只提供了一些對工程師而言最急切的功能，很多地方還有待進一步提升，未來我們會從以下幾個方面進行持續進行工作："}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對已有的功能進行持續優化完善，確保平臺穩定可靠"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"探索對分佈式事務的集成與支持，以對異常分佈式事務進行控制"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"提供配置中心功能，集中管理業務微服務配置"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"集成報警，讓工程師可以簡便快捷進行預警配置"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"支持功能快速擴展，當有新的功能需求時可以快速集成"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"對於微服務治理我們還是新人，未來我們仍將在這條路上持續學習、深入探索。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"作者介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"尚鵬飛"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"，FreeWheel高級研發工程師，任職於FreeWheel核心業務開發團隊，擅於解決後端業務系統的複雜問題，有豐富的開發經驗和敏捷團隊管理經驗，熱衷於新技術的探索與分享，目前致力於Golang微服務和系統重構相關工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]}]}