StreamNative 宣佈開源 Function Mesh： 簡化雲上的複雜流任務

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"StreamNative 鄭重宣佈開源 Function Mesh。Function Mesh 是爲事件流應用程序構建的無服務框架，爲在 Kubernetes 上運行的複雜事件流任務管理 "},{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/functions-overview\/","title":"","type":null},"content":[{"type":"text","text":"Pulsar Functions"}]},{"type":"text","text":" 和 "},{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/io-overview\/","title":"","type":null},"content":[{"type":"text","text":"Pulsar I\/O connector"}]},{"type":"text","text":"，增強應用程序的事件流功能。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Function Mesh 簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 是一種 Kubernetes operator，助力用戶在 Kubernetes 上原生使用 "},{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/functions-overview\/","title":"","type":null},"content":[{"type":"text","text":"Pulsar Functions"}]},{"type":"text","text":" 和 "},{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/io-overview\/","title":"","type":null},"content":[{"type":"text","text":"connectors"}]},{"type":"text","text":"，解鎖 Kubernetes 的全部特性，包括部署、擴縮容、管理應用程序等。例如，Function Mesh 依賴 Kubernetes 的調度能力，確保 Functions 的故障恢復能力，並且可以在任意時間適當調度 Functions。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 採用無服務架構，用於管理 Pulsar Functions 和 connectors，簡化了創建複雜流任務的流程。對於尋求雲原生無服務流解決方案的用戶而言，Function Mesh 是最佳選擇。Function Mesh 的主要優勢如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"便於管理多個 function 和 connector。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"充分使用 Kubernetes 調度器的全部功能，包括重平衡、重調度、容錯等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"完全解鎖 Pulsar Functions 和 connectors 在雲環境中的能力。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持在雲上的多個消息系統中使用 Pulsar Functions，或與雲環境中的現有工具進行集成（Function Mesh 與 Pulsar 相互獨立使用 Pulsar Functions 和 connectors）。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 適用於常見的輕量化流使用場景（如 ETL 任務），但不適合作爲流引擎單獨使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/58\/82\/58ee2800ea876dff52a0ccc9d6dab182.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"爲什麼使用 Function Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 2.0 版本，Pulsar 引入 Pulsar Functions 和 Pulsar I\/O connector。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/functions-overview\/","title":"","type":null},"content":[{"type":"text","text":"Pulsar Functions"}]},{"type":"text","text":" 是 Apache Pulsar 原生支持的無服務事件流架構。Pulsar Functions 支持用戶基於消息創建事件處理邏輯、簡化搭建事件流應用程序的操作、爲事件流引入無服務概念，從而避免部署單獨的系統。Pulsar Functions 的常見使用場景包括 ETL 任務、實時聚合、微服務、響應式服務、事件路由等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/io-overview\/","title":"","type":null},"content":[{"type":"text","text":"Pulsar I\/O connector"}]},{"type":"text","text":" 通過現有 Pulsar Functions 傳遞數據。Pulsar I\/O connector 包含兩個組件：source connector（即 source） 和 sink connector（即 sink）。Source 將外部系統中的數據寫入到 Pulsar；sink 則將 Pulsar 中的數據輸出到外部系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar Functions 和 Pulsar I\/O connector 簡化搭建事件流應用程序的操作。Pulsar Functions 支持在 Kubernetes 上運行 function 和 connector，但現有實現仍有一些不足。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Function 的元數據存儲在 Pulsar 中，但 function 的運行狀態由 Kubernetes 管理。這導致元數據和運行狀態之間可能出現不一致的情況，用戶管理 Pulsar Functions 變得困難。例如，當用戶從 Kuberbetes 中刪除運行 Pulsar Functions 的 StatefulSet 時，Pulsar 不會感知到該操作。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"現有的實現使用 Pulsar topic 存儲 function 元數據。如果存儲 function 元數據的 topic 臨時不可用，可能會造成 broker 故障循環。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"Functions 與特定 Pulsar 集羣綁定，因此很難跨集羣使用 function。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"目前，在 Kubernetes 上部署 Pulsar Functions 並實現特定功能（如自動擴縮容）有一定難度。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"越來越多的團隊選擇使用 Pulsar Functions 和 Pulsar I\/O connector 搭建無服務事件流應用程序，實現複雜的事件流能力。如果不借助 Function Mesh 等平臺，需要耗費大量人力來管理流任務中的多個 function 和 Pulsar I\/O connector。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決上述痛點問題，並使 Kubernetes 原生支持 Pulsar Functions 和 connector，簡化搭建複雜事件流任務的操作，我們開發了 Function Mesh。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"主要概念"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 支持用戶基於 Apache Pulsar 和其他流處理技術搭建事件流應用程序，其中的三個基本組件爲流、function 和 connector。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流是不可更改、僅追加的分區序列，用於存儲事件的歷史操作。例如，可以通過流事件爲金融交易建模，如“甲向乙發送 100 元”，然後“乙向丙發送 50 元”。流連接了 function 和 connector。Function Mesh 藉助 Apache Pulsar topic 實現流。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Function"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar Functions 是輕量級事件處理器，用於消費來自輸入流的消息。Pulsar Functions 將用戶提供的處理邏輯應用於接收到的消息，並將處理結果發送到其他流。Function Mesh 基於 Pulsar Functions 實現 function。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Connector"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Connector 用於爲流傳輸事件。Function Mesh 上有兩類 connector："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"source connector（即 source）：將外部系統中的事件寫入到流。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"sink connector（即 sink）：將流中的事件輸出到外部系統。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 基於 Pulsar I\/O connector 實現 connector。訪問 "},{"type":"link","attrs":{"href":"https:\/\/hub.streamnative.io\/","title":"","type":null},"content":[{"type":"text","text":"StreamNative Hub"}]},{"type":"text","text":" 官網，查看全部可用的 Pulsar I\/O connector。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"FunctionMesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FunctionMesh（即 Mesh）是多個流支撐的 function 和 connector 的集合（可以是有向無環圖（Directed Acyclic Graph，DAG），也可以是循環圖），可以實現強大的流處理邏輯。Function Mesh 支持同時創建、更新和終止多個 function 和 connector。同一個 Mesh 中， function 和 connector 的生命週期相同。Function Mesh 支持長期運行 function 和 connector，並且可以根據工作負載自動擴縮容 function 和 connector。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"Function Mesh 工作原理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh API 基於現有 Kubernetes API 實現，因此 Function Mesh 資源與其他 Kubernetes 原生資源兼容，集羣管理員可以使用現有 Kubernetes 工具管理 Function Mesh 資源。Function Mesh 採用 Kubernetes Custom Resource Definition（CRD），集羣管理員可以通過 CRD 自定義資源，開發事件流應用程序。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶可以使用 "},{"type":"codeinline","content":[{"type":"text","text":"kubectl"}]},{"type":"text","text":" CLI 工具將 CRD 直接提交到 Kubernetes 集羣，無需使用 "},{"type":"codeinline","content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具向 Pulsar 集羣發送 function 請求。Function Mesh 控制器監測 CRD 並創建 Kubernetes 資源，運行自定義的 function、source、sink 或 Mesh。這種方法的優勢在於 Kubernetes 直接存儲並管理 function 元數據和運行狀態，從而避免 Pulsar 現有方案中可能存在的元數據與運行狀態不一致問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖爲 Function Mesh 使用流程示意圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9b\/5c\/9bed9b674ab29cd6fbe59616ac95515c.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Function Mesh 架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 主要由 Kubernetes operator 和 Function Runner 兩個組件組成。Kubernetes operator 監測 Function Mesh CRD、創建 Kubernetes 資源（即 StatefulSet），從而在 Kubernetes 上運行 function、connector 和 Mesh。Function Runner 負責調用 function 和 connector 邏輯，處理從輸入流中接收的事件，並將處理結果發送到輸出流。目前，Function Runner 基於 Pulsar Functions runner 實現。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖爲 Function Mesh 的整體架構。當用戶創建 Function Mesh CRD 時，Function Mesh 控制器從 Kubernetes API 服務器接收已提交的 CRD，然後處理 CRD 並生成相應的 Kubernetes 資源。例如，Function Mesh 控制器在處理 Function CRD 時，會創建 StatefulSet，其上的每個 pod 都會啓動一個 Runner 來調用對應的 function。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/91\/5d\/91013bae635deb1f6dec0389157a3e5d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"如何使用 Function Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如需使用 Function Mesh，首先需要在 Kubernetes 集羣中安裝 Function Mesh operator 和 CRD。安裝 Function Mesh 的詳細信息，參閱"},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/install-function-mesh\/","title":"","type":null},"content":[{"type":"text","text":"安裝指南"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"安裝 Function Mesh operator 並部署 Pulsar 集羣后，用戶需要打包 function 和 connector，併爲 function、connector 和 Mesh 定義 CRD，再使用以下命令將 CRD 提交到 Kubernetes 集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"$ kubectl apply -f \/path\/to\/custom-crd.yaml \n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當 Kubernetes 集羣接收到 CRD 後，Function Mesh operator 逐一調度這些 CRD，並將 function 與其他必須的資源對象作爲 StatefulSet 運行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面我們將舉例解釋如何運行 function、connector 和 Mesh。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何使用 Function Mesh 運行 function"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 不會影響在雲上運行的 Pulsar Functions 的開發流程，但提交 function 時應使用 "},{"type":"codeinline","content":[{"type":"text","text":"yaml"}]},{"type":"text","text":" 文件而非 "},{"type":"codeinline","content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具。在後臺，我們開發了用於 Pulsar Functions 的 CRD 資源和相應的控制器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開發並測試 function 後，用戶需要打包 function 並將其提交到 Pulsar 集羣；或將其構建爲 Docker 鏡像並上傳到 image registry，再提交 function CRD 到 Pulsar 集羣。詳情參閱"},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/functions\/run-function\/run-java-function","title":"","type":null},"content":[{"type":"text","text":"使用 Function Mesh 運行 Pulsar Functions"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本示例使用 Function CRD 在 Kubernetes 集羣啓動 "},{"type":"codeinline","content":[{"type":"text","text":"ExclamationFunction"}]},{"type":"text","text":"，同時啓用自動擴縮容，並使用 Java 運行時與 Pulsar 消息系統進行交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: compute.functionmesh.io\/v1alpha1\nkind: Function\nmetadata:\n name: function-sample\n namespace: default\nspec:\n className: org.apache.pulsar.functions.api.examples.ExclamationFunction\n replicas: 1\n maxReplicas: 5\n image: streamnative\/function-mesh-example:latest\n logTopic: persistent:\/\/public\/default\/logging-function-logs\n input:\n topics:\n - persistent:\/\/public\/default\/source-topic\n typeClassName: java.lang.String\n output:\n topic: persistent:\/\/public\/default\/sink-topic\n typeClassName: java.lang.String\n resources:\n requests:\n cpu: \"0.1\"\n memory: 1G\n limits:\n cpu: \"0.2\"\n memory: 1.1G\n pulsar:\n pulsarConfig: \"test-pulsar\"\n java:\n jar: \"\/pulsar\/examples\/api-examples.jar\"\n"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何使用 Function Mesh 運行 connector"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Source 和 sink 是特定的 function。StreamNative 爲 Pulsar 內置和 StreamNative 託管的 connector 提供 Docker 鏡像。如果想要創建 Pulsar 內置或 StreamNative 託管的 connector，可以在 CRD 中指定 source 或 sink 的 Docker 鏡像。用戶可以在 Docker Hub 中找到相關的 Docker 鏡像，名稱格式爲 "},{"type":"codeinline","content":[{"type":"text","text":"streamnative\/pulsar-io-CONNECTOR-NAME:TAG"}]},{"type":"text","text":"，如 "},{"type":"codeinline","content":[{"type":"text","text":"streamnative\/pulsar-io-hbase:2.7.1"}]},{"type":"text","text":"。訪問 "},{"type":"link","attrs":{"href":"https:\/\/hub.streamnative.io\/","title":"","type":null},"content":[{"type":"text","text":"StreamNative Hub"}]},{"type":"text","text":" 官網即可查看 Function Mesh 支持的 connector。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用自定義 connector，則可以將其打包並上傳到 Pulsar Package Service 上，或構建成 Docker 鏡像再通過 CRD 提交。更多詳細信息，參閱"},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/connectors\/run-connector","title":"","type":null},"content":[{"type":"text","text":"使用 Function Mesh 運行 Pulsar connector"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在以下 source CRD yaml 示例文件中，connector 接收來自 "},{"type":"codeinline","content":[{"type":"text","text":"DebeziumMongoDB"}]},{"type":"text","text":" 的數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"DebeziumMongoDB connector yaml 文件："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: compute.functionmesh.io\/v1alpha1\nkind: Source\nmetadata:\n name: source-sample\nspec:\n image: streamnative\/pulsar-io-debezium-mongodb:2.7.1\n className: org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource\n replicas: 1\n output:\n topic: persistent:\/\/public\/default\/destination\n typeClassName: org.apache.pulsar.common.schema.KeyValue\n sourceConfig:\n mongodb.hosts: rs0\/mongo-dbz-0.mongo.default.svc.cluster.local:27017,rs0\/mongo-dbz-1.mongo.default.svc.cluster.local:27017,rs0\/mongo-dbz-2.mongo.default.svc.cluster.local:27017\n mongodb.name: dbserver1\n mongodb.user: debezium\n mongodb.password: dbz\n mongodb.task.id: \"1\"\n database.whitelist: inventory\n pulsar.service.url: pulsar:\/\/test-pulsar-broker.default.svc.cluster.local:6650\n pulsar:\n pulsarConfig: \"test-source\"\n java:\n jar: connectors\/pulsar-io-debezium-mongodb-2.7.1.nar\n jarLocation: \"\" # use pulsar provided connectors\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在以下 sink CRD yaml 示例文件中，connector 將數據發送到 "},{"type":"codeinline","content":[{"type":"text","text":"ElasticSearch"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"ElasticSearch connector yaml 文件："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: compute.functionmesh.io\/v1alpha1\nkind: Sink\nmetadata:\n name: sink-sample\nspec:\n image: streamnative\/pulsar-io-elastic-search:2.7.1\n className: org.apache.pulsar.io.elasticsearch.ElasticSearchSink\n replicas: 1\n input:\n topics:\n - persistent:\/\/public\/default\/input\n typeClassName: \"[B\"\n sinkConfig:\n elasticSearchUrl: \"http:\/\/quickstart-es-http.default.svc.cluster.local:9200\"\n indexName: \"my_index\"\n typeName: \"doc\"\n username: \"elastic\"\n password: \"X2Mq33FMWMnqlhvw598Z8562\"\n pulsar:\n pulsarConfig: \"test-sink\"\n java:\n jar: connectors\/pulsar-io-elastic-search-2.7.1.nar\n jarLocation: \"\" # use pulsar provided connectors\n"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"如何在 Kubernetes 上運行 Function Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FunctionMesh CRD 允許用戶定義一系列包含 function、source 和 sink 的資源，它們通過 "},{"type":"codeinline","content":[{"type":"text","text":"topics"}]},{"type":"text","text":" 字段連接起來，形成一個完整的數據流處理任務。當用戶提交自定義的 FunctionMesh CRD 到 Kubernetes 集羣后，FunctionMesh 控制器會協調 FunctionMesh CRD 中定義的多個 function、source 和 sink 資源，並將這些資源分配給相應的控制器，function、source和 sink 控制器則協調各個任務並啓動相應的 StatefulSet。FunctionMesh 控制器從系統收集每個 StatefulSet 的狀態，並整合爲 FunctionMesh 的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下示例中的 FunctionMesh 任務啓動了兩個 function，並通過這兩個 function 流式傳輸輸入，追加感嘆號。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: compute.functionmesh.io\/v1alpha1\nkind: FunctionMesh\nmetadata:\n name: mesh-sample\nspec:\n functions:\n - name: ex1\n className: org.apache.pulsar.functions.api.examples.ExclamationFunction\n replicas: 1\n maxReplicas: 5\n input:\n topics:\n - persistent:\/\/public\/default\/source-topic\n typeClassName: java.lang.String\n output:\n topic: persistent:\/\/public\/default\/mid-topic\n typeClassName: java.lang.String\n pulsar:\n pulsarConfig: \"mesh-test-pulsar\"\n java:\n jar: pulsar-functions-api-examples.jar\n jarLocation: public\/default\/test\n - name: ex2\n className: org.apache.pulsar.functions.api.examples.ExclamationFunction\n replicas: 1\n maxReplicas: 3\n input:\n topics:\n - persistent:\/\/public\/default\/mid-topic\n typeClassName: java.lang.String\n output:\n topic: persistent:\/\/public\/default\/sink-topic\n typeClassName: java.lang.String\n pulsar:\n pulsarConfig: \"mesh-test-pulsar\"\n java:\n jar: pulsar-functions-api-examples.jar\n jarLocation: public\/default\/test\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一個 function 的 output topic 和第二個 function 的 input topic 是同一個 topic。因此，當第一個 function 將結果發佈到其 output topic 中時，第二個 function 可以從該 topic 讀取數據。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用 "},{"type":"codeinline","content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用 Function Mesh operator 但不想改變創建、提交 function 的方式，則可以選擇 Function Mesh worker 服務。此服務類似於 Pulsar Functions worker 服務，使用 Function Mesh 調度並運行 function，並且支持用戶使用 "},{"type":"link","attrs":{"href":"https:\/\/pulsar.apache.org\/docs\/en\/pulsar-admin\/","title":"","type":null},"content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具管理 Function Mesh 中的 Pulsar Functions 和 connector。下圖展示了 Function Mesh worker 服務如何與 Pulsar proxy 協同合作，以及如何轉換並轉發請求到 Kubernetes 集羣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a1\/4e\/a12200caa3f7793fbf0ba0ccfa2aba4e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"詳情參閱"},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/install-function-mesh\/#work-with-pulsar-admin-cli-tool","title":"","type":null},"content":[{"type":"text","text":"使用 "},{"type":"codeinline","content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具"}]},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"遷移 Pulsar Functions 到 Function Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果用戶使用現有 Kubernetes 運行時運行 Pulsar Functions，並且想將這些 function 遷移到 Function Mesh，則可以通過遷移工具爲現有 function 生成 CRD 列表。然後，Function Mesh 通過這些 CRD 接管 Kubernetes 中 Pulsar Functions 的管理和運行。更多詳細信息，參閱"},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/migration\/migrate-function","title":"","type":null},"content":[{"type":"text","text":"遷移 Pulsar Functions 指南"}]},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"支持的特性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，Function Mesh 具有以下特性："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"確保 Kubernetes 原生支持 Pulsar Functions 和 connector。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"連接 Pulsar Functions 和 connector，組成流任務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與原 Pulsar Admin API 兼容，支持使用 "},{"type":"codeinline","content":[{"type":"text","text":"pulsar-admin"}]},{"type":"text","text":" CLI 工具提交 function 和 connector。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 Pod 水平自動擴縮（Horizontal Pod Autoscaler，HPA），按需自動擴縮 function 和 connector。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"身份驗證與授權。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Schema 和 SerDe。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持 Java、Python、Golang 等語言的運行時。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"未來規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新版本將會支持以下功能，如果您有任何建議或想爲 Function Mesh 做出貢獻，歡迎隨時與我們聯繫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提高 Function Mesh operator 的功能級別。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與 Pulsar Functions 功能匹敵。例如，實現有狀態的 function。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持基於已有 function 運行時的其他運行時，如 web-assembly。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"改進管理和檢測 Function Mesh 的工具\/前端。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲 function 單元分組，降低延遲，減少成本。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持基於 Pulsar metrics 的高級自動擴縮容。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將 function registry 與 "},{"type":"link","attrs":{"href":"http:\/\/pulsar.apache.org\/docs\/en\/next\/admin-api-packages\/","title":"","type":null},"content":[{"type":"text","text":"Apache Pulsar Packages"}]},{"type":"text","text":" 集成。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"開始試用 Function Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Function Mesh 已經"},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/function-mesh","title":"","type":null},"content":[{"type":"text","text":"開源"}]},{"type":"text","text":"，現在就在 Kubernetes 集羣上試用吧！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多關於 Function Mesh 的信息，參閱 "},{"type":"link","attrs":{"href":"https:\/\/functionmesh.io\/docs\/","title":"","type":null},"content":[{"type":"text","text":"Function Mesh 文檔"}]},{"type":"text","text":"或點擊觀看"},{"type":"link","attrs":{"href":"https:\/\/www.bilibili.com\/video\/BV1gy4y1W7iJ","title":"","type":null},"content":[{"type":"text","text":"操作演示"}]},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於 Function Mesh 的任何反饋或建議，可以通過"},{"type":"link","attrs":{"href":"mailto:[email protected]","title":"","type":null},"content":[{"type":"text","text":"電子郵件"}]},{"type":"text","text":"或在 GitHub 倉庫中"},{"type":"link","attrs":{"href":"https:\/\/github.com\/streamnative\/function-mesh","title":"","type":null},"content":[{"type":"text","text":"創建 issue"}]},{"type":"text","text":" 聯繫我們。"}]}]}