深度解讀：Apache DolphinScheduler 新架構與特性，性能提升2~3倍

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Apache DolphinScheduler(Incubating)是一個分佈式易擴展的可視化DAG工作流任務調度開源系統。\n解決數據研發ETL錯綜複雜的依賴關係，不能直觀監控任務健康狀態等問題。\nDolphinScheduler以DAG流式的方式將Task組裝起來，可實時監控任務的運行狀態，\n同時支持重試、從指定節點恢復失敗、暫停及Kill任務等操作"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"引言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"大數據任務調度作爲大數據建設中的核心基礎設施，在經過社區用戶們長期的使用中，不少用戶對調度也提出了很多新的要求，爲此，Apache DolphinScheduler(Incubating)經過長達  5 個多月的辛苦努力， 終於發佈了 1.3.2 正式版本。DolphinScheduler-1.3.2 有超過 30 名貢獻者參與開發，性能較 1.2 版本有 2 ~ 3 倍的提升，相對 1.2 版本，1.3.x 增加了諸如 K8s支持、多目錄管理等重要的新特性和新的任務類型。1.3.x 重要的改動如下："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"架構升級：重構 worker server"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"移除基於 ZooKeeper 的任務隊列"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"引入 Netty"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":" "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"進行 master 和 worker 間的通信"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"提供了三種 master 分發任務的算法："},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"隨機，輪詢和資源線性加權"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"Worker 不再操作數據庫，減輕數據庫壓力"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"新增任務類型"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"數據同步"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"節點：新增了 DataX 和 Sqoop 節點，加強 DolphinScheduler 構建全流程ETL 工作流的能力"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"條件分支"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"節點：提供了多個上游任務結果的複雜邏輯判斷能力，根據用戶自定義邏輯進行分支流轉"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"易用性提升"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"資源中心支持"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"目錄管理"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"：資源中心支持目錄類型，用戶可以分項目或模塊進行資源文件管理"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"支持 "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"Ambari 插件"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"：支持使用 Ambari 進行 DolphinScheduler 的集羣部署和管理"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"支持 "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"K8s"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"：DolphinScheduler 支持 K8s 部署。爲了支持 k8s，worker 分組數據不再存儲在 mysql，而通過配置文件中指定 worker 標籤的方式，存儲在 ZooKeeper 中"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"簡化配置文件：分離 install.sh 中的參數配置和集羣部署配置，install.sh 僅進行集羣部署，集羣參數配置文件抽取到 conf/config/install_config.conf 中"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"工作流佈局優化：提供一鍵美化工作流佈局功能"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"其他特性（部分）"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"增加工作流複製功能"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"刪除任務實例級聯刪除對應的任務實例日誌"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"1.3.x 新特性解讀"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"1、重構 Worker"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"DolphinScheduler 1.2 的整體架構"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/71/715dcd7eac49b7c367409db189dc2811.png","alt":null,"title":"DolphinScheduler 1.2 架構","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"在 DolphinScheduler 1.2 中，master 和 worker 的職責分別如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"master "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"master 採用分佈式無中心設計理念，master 主要負責 DAG 任務切分、任務提交監控，並同時監聽其它master 和 worker 的健康狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"master 服務啓動時向 Zookeeper 註冊臨時節點，通過監聽 Zookeeper 臨時節點變化來進行容錯處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該服務內主要包含:"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Distributed Quartz 分佈式調度組件，主要負責定時任務的啓停操作，當 quartz 調起任務後，Master內部會有線程池具體負責處理任務的後續操作"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MasterSchedulerThread 是一個掃描線程，定時掃描數據庫中的 command 表，根據不同的命令類型進行不同的業務操作"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MasterExecThread 主要是負責DAG任務切分、任務提交監控、各種不同命令類型的邏輯處理"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MasterTaskExecThread 主要負責任務的持久化"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"worker"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"worker 也採用分佈式無中心設計理念，worker 主要負責任務的執行和提供日誌服務。worker 服務啓動時向 Zookeeper 註冊臨時節點，並維持心跳。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該服務包含："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FetchTaskThread 主要負責不斷從 Task Queue 中領取任務，並根據不同任務類型調用TaskScheduleThread 對應執行器。"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"LoggerServer 是一個 RPC 服務，提供日誌分片查看、刷新和下載等功能"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"在 1.2 版本實現的架構中，任務隊列的實現基於 ZooKeeper。master 將任務數據存放到 ZooKeeper 中，然後 worker 節點通過分佈式鎖的方式去消費任務隊列，延遲了任務開始執行的時間。爲保證任務隊列的性能，ZooKeeper 的節點中並未存儲執行任務所需的全部數據。許多任務的元數據如租戶，隊列和任務實例信息等都需要由 worker 操作數據庫進行獲取，增加了數據庫的負擔。所以在 1.3 的架構設計中，我們着重考慮到減少 worker 的壓力，設計瞭如下新架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"DolphinScheduler 1.3 新架構"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/67/67035b3de288bdaf72028af5ce46af51.png","alt":null,"title":"DolphinScheduler 1.3 新架構","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"在 DolphinScheduler 1.3 中，任務隊列基於 Netty 實現，master 保留了原有的邏輯，當 master 節點切分出任務節點後，使用配置的任務分發策略直接發送目標 worker 節點進行執行。worker 節點在啓動的時候將節點信息和分組信息註冊到 ZooKeeper 中，供 master 節點進行調用。性能優化的核心是去除了 worker節點的 ZooKeeper 操作和數據庫操作。1.3 的架構分層詳細圖如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e1/e174e55c42e7d3cfe1d0b5a8ef62a7dd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"1.3 整個系統運作過程的活動圖如下："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/15/15bd043dbd1b80c5b197a908de83041b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"更多詳細信息參見 issue："},{"type":"link","attrs":{"href":"https://github.com/apache/incubator-dolphinscheduler/issues/1658","title":""},"content":[{"type":"text","text":"https://github.com/apache/incubator-dolphinscheduler/issues/1658"}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"2、新增任務類型"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"數據同步節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"DolphinScheduler 作爲一個數據處理調度系統支持了多種 ETL 功能節點，如 SQL 節點，存儲過程節點和 Spark 節點等。在整個 ETL 流程中，多源異構數據的集成是基礎。因此，在1.3.1版本中DolphinScheduler 集成了成熟的數據交換引擎 DataX 和 Sqoop 以支持多源異構數據源間的傳輸交換。目前，DolphinScheduler 已打通整個數據攝取-數據處理-數據結果同步的 ETL 流程。使用數據同步節點，可以避免在shell腳本中直接配置數據源的連接信息，所有的數據源權限均受 DolphinScheduler 管控。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"DataX 節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7f/7f458b19981618e111ad8777e59efefa.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"Sqoop節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/47/47914f5913513f299b6af7c9ba8ab723.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"條件分支節點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"Dolphin Scheduler 1.3.1 支持條件分支節點，用戶可以在自定義參數中定義分支流轉的判斷邏輯，根據上游任務的執行情況，決定後續執行的分支。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f7/f7ec3f62fddd68c361a2f26b497d081f.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"注意，條件節點是一種邏輯判斷任務，不會分發到 worker 去執行，是在 master 上執行的一個邏輯節點。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"3、資源中心支持目錄結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"DolphinScheduler 1.3.1 支持了資源中心目錄化和授權資源使用目錄樹的功能，極大的改善了資源文件的使用體驗。資源中心目錄化使得分項目管理資源文件成爲可能而不是將租戶的所有資源文件都放在一個目錄下。資源中心支持授權整個目錄給用戶，提高了多文件授權的操作效率。使用目錄樹的方式進行資源文件授權，可以避免 1.2 版本一個一個尋找資源文件的耗時操作。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/db/dbf7d88bba4ad04dd97abd5731a78491.png","alt":"image.png","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9b/9bef91bb40e8dbeeadf2f320d98c72a3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"4、支持 Ambari 插件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"Ambari 插件可以讓 DolphinScheduler 和 Ambari 輕鬆集成，利用 Ambari 的能力可以使部署和管理 DolphinScheduler 更加簡單，也更容易擴/縮容，"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/31/313be423d45db1733ee81c6ab7240cff.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b9/b990fe93d30668df2f8036df82c64534.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"5、支持 K8s"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"1.3.1 也提供了對 K8s 的支持，後續也會推出 DolphinScheduler 的 Docker 官方鏡像，更方便大家部署 DolphinScheduler，這塊的詳細文檔請參考：[ K8s Readme ]"},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}},{"type":"strong"}],"text":"("},{"type":"link","attrs":{"href":"https://github.com/apache/incubator-dolphinscheduler/blob/1.3.1-release/docker/kubernetes/dolphinscheduler/README.md","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://github.com/apache/incubator-dolphinscheduler/blob/1.3.1-release/docker/kubernetes/dolphinscheduler/README.md"}],"marks":[{"type":"strong"}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}},{"type":"strong"}],"text":")"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e4/e4baed1e972590ee616048d2f56e54b3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"6、其他一些重要的特性："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"批量導出和導入工作流"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"流程定義複製"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"刪除流程實例級聯刪除任務日誌"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"DAG 圖一鍵格式化，非常適合通過 open API 調用的場景"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"流程圖美化"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"簡化配置，優化部署體驗"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#2e2e2e","name":"user"}},{"type":"strong"}],"text":"完善自動化 CI、CD"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.3.2 版本帶來的新特性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"新特性："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Worker Server 可以設置多個 worker groups"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"JVM 參數優化"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"給流程圖連線添加標籤"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"值得注意的是 1.3.2 版本修復了 1.3.1 的 20 多個 bug ，其中需要關注的是"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"["},{"type":"link","attrs":{"href":"https://github.com/apache/incubator-dolphinscheduler/releases/tag/1.3.2","title":""},"content":[{"type":"text","text":"#3058"}]},{"type":"text","text":"] The task running order in the process instance does not follow the topological order in the process definition"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 該 bug 是在打開任務節點，沒有正常關閉窗口，然後又新建一個流程定義導致新建的流程定義的前置節點的信息丟失。這可能造成 1.3.1 版本的流程圖運行錯亂的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，1.3.2 也帶來了 4 項功能改進和增強，具體變化請參考："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/apache/incubator-dolphinscheduler/releases/tag/1.3.2","title":""},"content":[{"type":"text","text":"https://github.com/apache/incubator-dolphinscheduler/releases/tag/1.3.2"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"Apache DolphinScheduler社區介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache DolphinScheduler 是一個非常多樣化的社區，至今貢獻者已120多名， 他們分別來自 30 多家不同的公司。 微信羣用戶3000人。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/80/803ed5aa199c7eef26cce083113d7087.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":"部分用戶案例(排名不分先後)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"已經有近 400 多家企業和科研機構在使用 DolphinScheduler，來處理各類調度和定時任務，另有 800 多家公司開通了海豚調度的試用："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/cb/cb612d22a7484b1dcc5e7fde5d0404a9.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Apache DolphinScheduler 能力"}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以DAG圖的方式將Task按照任務的依賴關係關聯起來，可實時可視化監控任務的運行狀態"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持豐富的任務類型：Shell、MR、Spark、Flink、SQL(mysql、postgresql、hive、sparksql)、Python、Http、Sub_Process、Procedure等"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持工作流定時調度、依賴調度、手動調度、手動暫停/停止/恢復，同時支持失敗重試/告警、從指定節點恢復失敗、Kill任務等操作"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持工作流優先級、任務優先級及任務的故障轉移及任務超時告警/失敗"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持工作流全局參數及節點自定義參數設置"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持資源文件的在線上傳/下載，管理等，支持在線文件創建、編輯"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持任務日誌在線查看及滾動、在線下載日誌等"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現集羣HA，通過Zookeeper實現Master集羣和Worker集羣去中心化"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持對"},{"type":"codeinline","content":[{"type":"text","text":"Master/Worker"}]},{"type":"text","text":" cpu load，memory，cpu在線查看"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持工作流運行歷史樹形/甘特圖展示、支持任務狀態統計、流程狀態統計"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持補數"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持多租戶"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持國際化"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"加入 Apache DolphinScheduler"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在使用 DolphinScheduler 的過程中，如果您有任何問題或者想法、建議，都可以通過Apache 郵件列表或者github issue參與到 DolphinScheduler 的社區建設中來。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎加入貢獻的隊伍，加入開源社區從提交第一個 PR開始,"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  - 找到帶有”easy to fix”標記或者一些非常簡單的issue(比如拼寫錯誤等)，先通過第一個PR熟悉提交流程，如果有任何疑問，歡迎聯繫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"郵件訂閱方式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://dolphinscheduler.apache.org/zh-cn/docs/development/subscribe.html","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://dolphinscheduler.apache.org/zh-cn/docs/development/subscribe.html"}],"marks":[{"type":"strong"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"github："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://github.com/apache/incubator-dolphinscheduler","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://github.com/apache/incubator-dolphinscheduler"}],"marks":[{"type":"strong"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}},{"type":"strong"}],"text":"官方網站"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://dolphinscheduler.apache.org/en-us/","title":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"https://dolphinscheduler.apache.org/en-us/"}],"marks":[{"type":"strong"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#262626","name":"user"}}],"text":"歡迎下載試用，在試用過程中發現任何問題，可以通過郵件列表或 github 上新建 issue 進行反饋！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}