dolphinscheduler整合clickhouse數據源-挖坑

dolphinscheduler版本2.0.0

測試步驟:

1.新建數據源測試成功

2.新建工作流執行失敗日誌(從日誌看沒有看錯什麼異常)

[LOG-PATH]: /var/log/udp/1.0.0.0/dolphinscheduler/8136281159680_2/2773/4914.log, [HOST]:  10.30.10.62
2023-03-21 16:20:38.046 [INFO]  - [taskAppId=TASK-8136281159680_2-2773-4914]:[119] - Full sql parameters: SqlParameters{type='CLICKHOUSE', datasource=7, sql='select * from ddp_fat_dwh.temp limit 10
', sqlType=0, sendEmail=false, displayRows=10, limit=0, udfs='', showType='null', connParams='', groupId='0', title='', preStatements=[], postStatements=[]}
2023-03-21 16:20:38.052 [INFO]  - [taskAppId=TASK-8136281159680_2-2773-4914]:[120] - sql type : CLICKHOUSE, datasource : 7, sql : select * from ddp_fat_dwh.temp limit 10
 , localParams : [],udfs : ,showType : null,connParams : ,varPool : [] ,query max result limit  0
2023-03-21 16:20:38.053 [INFO]  - [taskAppId=TASK-8136281159680_2-2773-4914]:[443] - after replace sql , preparing : select * from ddp_fat_dwh.temp limit 10

2023-03-21 16:20:38.053 [INFO]  - [taskAppId=TASK-8136281159680_2-2773-4914]:[452] - Sql Params are replaced sql , parameters:
2023-03-21 16:20:38.053 [INFO]  - [taskAppId=TASK-8136281159680_2-2773-4914]:[522] - can't find udf function resource

3.去對應的work節點查看日誌,具體日誌路徑以爲work的log配置文件爲準:/var/log/udp/1.0.0.0/dolphinscheduler

2023-03-21 16:52:22.958 [INFO]  - [taskAppId=TASK-8136281159680_2-2771-4915]:[443] - after replace sql , preparing : SELECT 1;
2023-03-21 16:52:22.958 [INFO]  - [taskAppId=TASK-8136281159680_2-2771-4915]:[452] - Sql Params are replaced sql , parameters:
2023-03-21 16:52:22.959 [INFO]  - [taskAppId=TASK-8136281159680_2-2771-4915]:[522] - can't find udf function resource
2023-03-21 16:52:22.960 [ERROR] org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[210] - task scheduler failure
java.lang.NoClassDefFoundError: org/apache/http/conn/ConnectionKeepAliveStrategy
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:73)
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55)
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47)
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at org.apache.dolphinscheduler.plugin.task.datasource.clickhouse.ClickHouseDatasourceProcessor.getConnection(ClickHouseDatasourceProcessor.java:102)
	at org.apache.dolphinscheduler.plugin.task.datasource.DatasourceUtil.getConnection(DatasourceUtil.java:59)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:184)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:155)
	at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:192)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.ConnectionKeepAliveStrategy
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at org.apache.dolphinscheduler.spi.plugin.DolphinPluginClassLoader.loadClass(DolphinPluginClassLoader.java:81)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 16 common frames omitted

看到具體的異常日誌大概知道原因了,需要將相關的依賴上傳到work節點對應的lib目錄/srv/udp/1.0.0.0/dolphinscheduler/lib/plugin/task/sql

4.clickhouse的依賴很多將所有的依賴打包輸出

4.1新建一個maven項目添加clickhouse依賴

4.2執行命令mvn dependency:copy-dependencies -DoutputDirectory=C:\\my\\out

4.3.將依賴上傳並且授權

 

5.重啓work節點搞定!!!

 

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章