cdh6.3.2配置Hive on Spark

環境：戴爾xps15（32G內存、1T固態、外接雷電3接口的三星1T移動固態、WD Elements的4T外接機械硬盤）win10三臺Centos7虛擬機用於測試cdh6.3.2集羣（免費的社區版的最高版本）以及自編譯phoenix5.1.0、flink1.10.0、elasticsearch6.6.0等各源碼。

首先，該集羣試用企業版功能到期後，無法啓動Reports Manager角色（見圖一）且Alert Publisher角色部分功能被閹割（見圖二）。

1.1、添加服務以搭建Hive on Spark，由於筆記本上虛擬機的cdh集羣角色易掛故HDFS配置爲HA高可用模式。

1.2、Hive服務的“配置”處搜索spark（見下圖）勾選兩配置項，且圖中帶藍色回退小按鈕的參數項爲我調試成功後的最終配置（注：各內存若不修改默認值，則初次運行hive時各種報錯）

1.3、在上圖勾選spark時會提示要配置Spark On Yarn，到spark服務的“配置”處確認一下

1.4、Yarn服務“配置”處第10頁修改默認資源值，可避免下面所列報錯

其中yarn.nodemanager.resource.memory-mb和yarn.scheduler.maximum-allocation-mb爲必修項（由默認值1改爲2）。而yarn.nodemanager.resource.cpu-vcores和yarn.scheduler.maximum-allocation-vcores由默認值2修改爲4是爲了解決yarn列表中如下報錯

1.5、開始運行hive:

1.5.1、若直接運行hive進入其客戶端，弊端爲在一開始無法初始化spark client更別提提交到yarn時無法看到詳細日誌（如下圖）

本來調試階段可以按照Hive命令行中進行設置，無奈始終並不打印日誌（見圖），權限和報錯問題

INFO [spark-submit-stderr-redir-6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] client.SparkClientImpl: Exception in                                       thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergr                                      oup:drwxr-xr-x
------------省略------------------------
ERROR [6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] client.SparkClientImpl: Error while waiting for Remote Spark D                                      river to connect back to HiveServer2.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ?
---------------------
ERROR [6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] spark.SparkTask: Failed to execute Spark task "Stage-1"
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 6f70c261-ed5e-4af4-a7ea-7b11084e7547_0                                      : java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ?
--------
Caused by: java.lang.RuntimeException: Error while waiting for Remote Spark Driver to connect back to HiveServer2.

於是把hive運行命令改爲

圖中所說的無果是指hive.log文件中就一行hive-site.xml日誌，無其他任何日誌。不過控制檯卻打印出來很多有用日誌

關於此“超時”報錯的詳解見鏈接，經嘗試由默認值60s改爲200s解決超時報錯

set hive.spark.client.future.timeout=200;
可解決報錯：
ERROR client.SparkClientImpl: Timed out waiting for Remote Spark Driver to connect to HiveServer2.
Possible reasons include network issues, errors in remote driver, cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remoark Driver
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:103)
        at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:90)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:104)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:100)
        at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:77)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:132)
        at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:122)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:178)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)
20/05/02 23:37:26 ERROR spark.SparkTask: Failed to execute Spark task "Stage-1"
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session b1fb6956-93d2-438a-98c2-a9db64024ac4_1: java.util.concurrent.TimeoutExceptioient 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:286)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:135)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:132)
        at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:122)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.RuntimeException: Timed out waiting for Remote Spark Driver to connect to HiveServer2.
Possible reasons include network issues, errors in remote driver, cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:124)
        at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:90)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:104)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:100)
        at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:77)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:131)
        ... 22 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection fre Remote Spark Driver
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:103)
        ... 27 more
Caused by: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:178)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session b1fb6956-93d2-438a-98c2-a9db644_1: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
20/05/02 23:37:26 ERROR ql.Driver: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark sesb1fb6956-93d2-438a-98c2-a9db64024ac4_1: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remotrk Driver

只不過此參數不可在sql同命令行中設置（見下圖）